442 Commits (2b25fdb80c00b19803b148abbd3faab76a40132c)

Author	SHA1	Message	Date
Corey Farrell	d893e57c90	Fix GCC 8 build issues. ... This fixes build warnings found by GCC 8. In some cases format truncation is intentional so the warning is just suppressed. ASTERISK-27824 #close Change-Id: I724f146cbddba8b86619d4c4a9931ee877995c84	7 years ago
Corey Farrell	6f304697b0	core: Remove dead symbols from asterisk.exports.in. ... * dahdi_chan_name * dahdi_chan_name_len * dahdi_chan_mode * __manager_event * dialed_interface_info Added comment about __progname and environ being needed for FreeBSD to prevent accidental removal in the future. Change-Id: I3ae026bc541cd9cb572be2ffa95fc359547642b5	7 years ago
Sean Bright	ce3d56920b	Remove as much trailing whitespace as possible. ... Change-Id: I873c1c6d00f447269bd841494459efccdd2c19c0	8 years ago
Corey Farrell	82b6ba976f	Fix Common Typo's. ... Fix instances of: * Retreive * Recieve * other then * different then * Repeated words ("the the", "an an", "and and", etc). * othterwise, teh ASTERISK-24198 #close Change-Id: I3809a9c113b92fd9d0d9f9bac98e9c66dc8b2d31	8 years ago
Corey Farrell	501f4dcdd8	aco: Minimize use of regex. ... Remove nearly all use of regex from ACO users. Still remaining: * app_confbridge has a legitamate use of option name regex. * ast_sorcery_object_fields_register is implemented with regex, all callers use simple prefix based regex. I haven't decided the best way to fix this in both 13/15 and master. Change-Id: Ib5ed478218d8a661ace4d2eaaea98b59a897974b	8 years ago
Corey Farrell	115f53a0fb	CLI: Remove calls to ast_cli_generator. ... The ability to add to localized storage cannot be supported by ast_cli_generator. The only calls to ast_cli_generator should be by functions that need to proxy the CLI generator, for example 'cli check permissions' or 'core show help'. * ast_cli_generatornummatches now retrieves the vector of matches and reports the number of elements (not including 'best' match). * test_substitution retrieves and iterates the vector. Change-Id: I8cd6b93905363cf7a33a2d2b0e2a8f8446d9f248	8 years ago
Joshua Colp	4e6a0d5683	Merge "menuselect: Remove ineffective weak attribute detection." into 13	8 years ago
Corey Farrell	5fe2e7bfdc	tests: Fix warnings found on Mac. ... test_pbx used raise without explicitly including signal.h. On Mac for some reason nothing else includes it. test_logger checked if an unsigned int was negative. Switch the variable to 'int' so that error check can be effective. Change-Id: Ie1db5dd1818ac25cc2ae41b644f848b5865b1362	8 years ago
Corey Farrell	455b1bdeb0	menuselect: Remove ineffective weak attribute detection. ... menuselect detects compiler support for multiple styles of weak functions. This is a remnant from 2013 when OPTIONAL_API required weak functions. It is no longer correct for menuselect to switch dependencies from optional to required based on lack of weak function support. Note an issue remains - dependencies should switch from optional to required based on OPTIONAL_API being enabled or disabled. I don't think this is possible. menuselect needs to know at startup if OPTIONAL_API is enabled or disabled, so the only way to fix this is to remove OPTIONAL_API from menuselect and create a configure option. I've left the code that switches in place but it's preprocessed out. Additionally removed: - WEAKREF variable from Asterisk makeopts.in. - Related disabled code from test_utils. - Pointless AC_REVISION call from menuselect/configure.ac. Change-Id: Ifa702e5f98eb45f338b2f131a93354632a8fb389	8 years ago
Corey Farrell	8cdc0ef385	test_sorcery_memory_cache_thrash: Handle error from vector append. ... Cleanup resources when we fail to append the vector and report test failure. Change-Id: I6eb41586fd11dee8c0dfe35e91cb465a4cab7298	8 years ago
Corey Farrell	7c35740ba1	Add missing menuselect dependencies. ... This adds menuselect dependencies for modules that use symbols of other modules. ASTERISK-27390 Change-Id: Ia2d2849f5b87a72af7324a82edc3f283eafb5385	8 years ago
Corey Farrell	6cec3149be	test_config: Fix failure and segfault when config_hook is run twice. ... On second run the config_hook test was unexpectedly failing to load test_config.conf because it was still unmodified since the last load. This is fixed by not passing CONFIG_FLAG_FILEUNCHANGED for the initial loads, only using it when we are tested that a reload of unmodified files do not initiate the hook. ASTERISK-25960 Change-Id: Ifd679509a23ed163e5cc647490bf7df4ae3cd856	8 years ago
Corey Farrell	8bf4be1048	vector: multiple evaluation of elem in AST_VECTOR_ADD_SORTED. ... Use temporary variable to prevent multiple evaluations of elem argument. This resolves a memory leak in res_pjproject startup. ASTERISK-27317 #close Change-Id: Ib960d7f5576f9e1a3c478ecb48995582a574e06d	8 years ago
Corey Farrell	84b6a5efd7	Correct some leaks in unit tests. ... * chan_sip: channel in test_sip_rtpqos_1. * test_config: config hook, config info and global config holder. * test_core_format: format in format_attribute_set_without_interface. * test_stream: unneeded frame duplication. * test_taskprocessor: task_data. Change-Id: I94d364d195cf3b3b5de2bf3ad565343275c7ad31	8 years ago
Corey Farrell	df49ad2528	core: Add PARSE_TIMELEN support to ast_parse_arg and ACO. ... This adds support for parsing timelen values from config files. This includes support for all flags which apply to PARSE_INT32. Support for this parser is added to ACO via the OPT_TIMELEN_T option type. Fixes an issue where extra characters provided to ast_app_parse_timelen were ignored, they now cause an error. Testing is included. ASTERISK-27117 #close Change-Id: I6b333feca7e3f83b4ef5bf2636fc0fd613742554	8 years ago
Joshua Colp	e414833f6e	bridge: Add a deferred queue. ... This change adds a deferred queue to bridging. If a bridge technology determines that a frame can not be written and should be deferred it can indicate back to bridging to do so. Bridging will then requeue any deferred frames upon a new channel joining the bridge. This change has been leveraged for T.38 request negotiate control frames. Without the deferred queue there is a race condition between the bridge receiving the T.38 request negotiate and the second channel joining and being in the bridge. If the channel is not yet in the bridge then the T.38 negotiation fails. A unit test has also been added that confirms that a T.38 request negotiate control frame is deferred when no other channel is in the bridge and that it is requeued when a new channel joins the bridge. ASTERISK-26923 Change-Id: Ie05b08523f399eae579130f4a5f562a344d2e415	8 years ago
George Joseph	c10341646d	test_json: Fix test names with reserved words ... Some of the test names were actually reserved words (true, false, int, null, string, bool). When the jenkins test results analyzer does its thing it tries to create a map using the test names as keys and fails because they're reserved words. Added "type_" to those test names. Change-Id: I90d809f46969c78a1c605b736ff0635196a2cf1b	8 years ago
George Joseph	65898c3af8	unittests: Add a unit test that causes a SEGV and... ... ...that can only be run by explicitly calling it with 'test execute category /DO_NOT_RUN/ name RAISE_SEGV' This allows us to more easily test CI and debugging tools that should do certain things when asterisk coredumps. To allow this a new member was added to the ast_test_info structure named 'explicit_only'. If set by a test, the test will be skipped during a 'test execute all' or 'test execute category ...'. Change-Id: Ia3a11856aae4887df9a02b6b081cc777b36eb6ed	8 years ago
George Joseph	7d4a22bf2e	logger: Added logger_queue_limit to the configuration options. ... All log messages go to a queue serviced by a single thread which does all the IO. This setting controls how big that queue can get (and therefore how much memory is allocated) before new messages are discarded. The default is 1000. Should something go bezerk and log tons of messages in a tight loop, this will prevent memory escalation. When the limit is reached, a WARNING is logged to that effect and messages are discarded until the queue is empty again. At that time another WARNING will be logged with the count of discarded messages. There's no "low water mark" for this queue because the logger thread empties the entire queue and processes it in 1 batch before going back and waiting on the queue again. Implementing a low water mark would mean additional locking as the thread processes each message and it's not worth it. A "test" was added to test_logger.c but since the outcome is non-deterministic, it's really just a cli command, not a unit test. Change-Id: Ib4520c95e1ca5325dbf584c7989ce391649836d1	8 years ago
Kevin Harwell	4d3b4fbf22	vector: defaults and indexes ... Added an pre-defined integer vector declaration. This makes integer vectors easier to declare and pass around. Also, added the ability to default a vector up to a given size with a default value. Lastly, added functionality that returns the "nth" index of a matching value. Also, updated a unit test to test these changes. Change-Id: Iaf4b51b2540eda57cb43f67aa59cf1d96cdbcaa5	8 years ago
George Joseph	f882ca2572	modules: change module LOAD_FAILUREs to LOAD_DECLINES ... In all non-pbx modules, AST_MODULE_LOAD_FAILURE has been changed to AST_MODULE_LOAD_DECLINE. This prevents asterisk from exiting if a module can't be loaded. If the user wishes to retain the FAILURE behavior for a specific module, they can use the "require" or "preload-require" keyword in modules.conf. A new API was added to logger: ast_is_logger_initialized(). This allows asterisk.c/check_init() to print to the error log once the logger subsystem is ready instead of just to stdout. If something does fail before the logger is initialized, we now print to stderr instead of stdout. Change-Id: I5f4b50623d9b5a6cb7c5624a8c5c1274c13b2b25	8 years ago
George Joseph	bcb2c1f62f	Merge "tests: use datadir for sound files" into 13	8 years ago
George Joseph	d6f67df8ce	Merge "test_voicemail_api: order of params to VERIFY macros" into 13	8 years ago
George Joseph	17f4989d49	ari: Implement 'debug all' and request/response logging ... The 'ari set debug' command has been enhanced to accept 'all' as an application name. This allows dumping of all apps even if an app hasn't registered yet. To accomplish this, a new global_debug global variable was added to res/stasis/app.c and new APIs were added to set and query the value. 'ari set debug' now displays requests and responses as well as events. This required refactoring the existing debug code. * The implementation for 'ari set debug' was moved from stasis/cli.{c,h} to ari/cli.{c,h}, and stasis/cli.{c,h} were deleted. * In order to print the body of incoming requests even if a request failed, the consumption of the body was moved from the ari stubs to ast_ari_callback in res_ari.c and the moustache templates were then regenerated. The body is now passed to ast_ari_invoke and then on to the handlers. This results in code savings since that template was inserted multiple times into all the stubs. An additional change was made to the ao2_str_container implementation to add partial key searching and a sort function. The existing cli code assumed it was already there when it wasn't so the tab completion was never working. Change-Id: Ief936f747ce47f1fb14035fbe61152cf766406bf	8 years ago
Tzafrir Cohen	cd2677f966	tests: use datadir for sound files ... Some (voicemail-related) tests API symlinks beep.gsm and other files from ast_config_AST_VAR_DIR. It should use ast_config_AST_DATA_DIR. ASTERISK-26740 #close Change-Id: Id49c56fb9e16df64b1a2b829693ca7601252df89	8 years ago
Tzafrir Cohen	b62f84bfb1	test_voicemail_api: order of params to VERIFY macros ... Fix order of parameters in calls to VM_API_INT_VERIFY and VM_API_STRING_VERIFY ASTERISK-26739 #close Change-Id: I30dc6b36893aadad6012be3f16f93aa5720870d6 Note: status: builds. Not tested any further.	8 years ago
Richard Mudgett	a969bf3577	LISTFILTER: Remove outdated ERROR message. ... Feeding LISTFILTER an empty variable results in an invalid ERROR message. Earlier changes made the message useless because we can no longer tell if the variable is empty or does not exist. It is valid to try to remove a value from an empty list just as it is valid to try to remove a value that is not in a non-empty list. * Removed the outdated ERROR message. * Added more test cases to the LISTFILTER unit test. Change-Id: Ided9040e6359c44a335ef54e02ef5950a1863134	8 years ago
George Joseph	c92dcc76da	file.c/__ast_file_read_dirs: Fix issues on filesystems without d_type ... One of the code paths in __ast_file_read_dirs will only get executed if the OS doesn't support dirent->d_type OR if the filesystem the particular file is on doesn't support it. So, while standard Linux systems support the field, some filesystems like XFS do not. In this case, we need to call stat() to determine whether the directory entry is a file or directory so we append the filename to the supplied directory path and call stat. We forgot to truncate path back to just the directory afterwards though so we were passing a complete file name to the callback in the dir_name parameter instead of just the directory name. The logic has been re-written to only create a full_path if we need to call stat() or if we need to descend into another directory. Change-Id: I54e4228bd8355fad65200c6df3ec4c9c8a98dfba	9 years ago
Kevin Harwell	bd4d7d8ad0	stasis_recording/stored: remove calls to deprecated readdir_r function. ... The readdir_r function has been deprecated and should no longer be used. This patch removes the readdir_r dependency (replaced it with readdir) and also moves the directory search code to a more centralized spot (file.c) Also removed a strict dependency on the dirent structure's d_type field as it is not portable. The code now checks to see if the value is available. If so, it tries to use it, but defaults back to using the stats function if necessary. Lastly, for most implementations of readdir it *should* be thread-safe to make concurrent calls to it as long as different directory streams are specified. glibc falls into this category. However, since it is possible that there exist some implementations that are not safe, locking has been added for those other than glibc. ASTERISK-26412 ASTERISK-26509 #close Change-Id: Id8f54689b1e2873e82a09d0d0d2faf41964e80ba	9 years ago
George Joseph	3a2092b722	test_astobj2_thrash: Fix multithreaded issues ... The test uses 4 threads to grow, count, lookup and shrink 15K objects in a container. If there's only 1 execution engine available, the test will complete in <50ms. If each threads gets its own execution engine, the test may timeout after 60 seconds because the count thread does a locked ao2_callback on the whole container in a tight loop with only a sched_yield to give up time. The lock contention makes the test execution times wildly variable and mostly timeout. 2 execution engines are OK, 3 results in about 33% failure rate and >=4 causes a 80% failure rate. To fix, the sched_yield was changed to a usleep(500). Also, the number of buckets specified for the container was an even number so that was changed to the next prime number greater than (MAX_HASH_ENTRIES / 100). That's 151 currently. Change-Id: I50cd2344161ea61bfe4b96d2a29a6ccf88385c77	9 years ago
Matt Jordan	42cfdcd1b7	res/ari: Add the Asterisk EID field to outgoing events ... This patch adds the Asterisk EID field to all outgoing ARI events. Because this field should be added to all events as they are transmitted, it is appended to the JSON message just prior to it being handed off to the application message handler. This makes it somewhat resilient to both new events being added to ARI, as well as other potential event transport mechanisms. ASTERISK-26470 #close Change-Id: Ieff0ecc24464e83f3f44e9c3e7bd9a5d70b87a1d	9 years ago
Richard Mudgett	7f8f125738	json: Check party id name, number, subaddresses for UTF-8. ... * Updated unit test as ast_json_name_number() is now NULL tolerant. ASTERISK-26466 #close Reported by: Richard Mudgett Change-Id: I7d4e14194f8f81f24a1dc34d1b8602c0950265a6	9 years ago
Torrey Searle	6f448f32fe	res_ari: Add http prefix to generated docs ... updated the uri handler to include the url prefix of the http server this enables res_ari to add it to the uris when generating docs Change-Id: I279335a2625261a8492206c37219698f42591c2e	9 years ago
Alexei Gradinari	b4a9fa2c9e	res_sorcery_realtime: fix bug when successful UPDATE is treated as failed ... If the SQL UPDATE statement changes nothing then SQLRowCount returns 0. This value should be treated as success. But the function sorcery_realtime_update treats it as failed. This bug was found using stress tests on PJSIP. If there are 2 consecutive SIP REGISTER requests with the same contact data during 1 second then res_pjsip_registrar adds contact location on 1st request and tries to update contact location on 2nd. The update fails and res_pjsip_registrar even removes correct contact location. The test "object_update_uncreated" was removed from test_sorcery_realtime.c because it's now a valid situation. This patch also adds missing debug of extra SQL parameter. ASTERISK-26172 #close Change-Id: I05a7f3051455336c9dda29efc229decf86071303	9 years ago
George Joseph	77da168e58	test_res_pjsip_scheduler: Add 'depends' on pjproject in MODULEINFO ... Since the file was missing the depends on pjproject, it wasn't picking up the pjproject related include path. If there was no system installed pjproject and pjproject-bundled was used, a compile would fail because pjsip.h wasn't found. ASTERISK-26139 #close Change-Id: I2ee64a999051452bc198c4e2c168c70769cd3757	9 years ago
Joshua Colp	c2f68787a3	Merge "cel: Ensure only one dial status per channel exists." into 13	9 years ago
Joshua Colp	caf6cccc5c	cel: Ensure only one dial status per channel exists. ... CEL wrongly assumed that a channel would only have a single dial event on it. This is incorrect. Particularly in a queue each call attempt to a member will result in a dial event, adding a new dial status in CEL without removing the old one. This would cause the container to grow with only one dial status being removed when the channel went away. The other dial status entries would remain leaking memory. This change fixes the memory leak by ensuring that only one dial status will only ever exist for each channel. The behavior during the scenario where multiple events are received has also been improved. For failure cases the first failure will be the dial status. If an answer dial status is received, though, it will take priority and the dial status for the channel will be answer. Memory usage has also been decreased by storing the minimal amount of information and the code has been cleaned up slightly. ASTERISK-25262 #close Change-Id: I5944eb923db17b6a0faa7317ff6abc9307c009fe	9 years ago
George Joseph	a99ddc6a0d	build: Fix ast_sockaddr initialization to be more portable ... A change to glibc 2.22 changed the order of the sockadddr_storage members which caused the places where we do an initialization of ast_sockaddr with '{ { 0, 0, } }' to fail compilation. Those initializers (which we shouldn't have been using anyway) have been replaced with memsets. Change-Id: Idd1b3b320903d8771bfe221f0b015685de628fa4	9 years ago
Joshua Colp	a16aa467f5	Merge "config: Fix ast_config_text_file_save2 writability check for missing files" into 13	9 years ago
Joshua Colp	e5c5fac8f1	Merge "res_pjsip: Add serialized scheduler (res_pjsip/pjsip_scheduler.c)" into 13	9 years ago
George Joseph	b38f1146e5	config: Fix ast_config_text_file_save2 writability check for missing files ... A patch I did back in 2014 modified ast_config_text_file_save2 to check the writability of the main file and include files before truncating and re-writing them. An unintended side-effect of this was that if a file doesn't exist, the check fails and the write is aborted. This patch causes ast_config_text_file_save2 to check the writability of the parent directory of missing files instead of checking the file itself. This allows missing files to be created again. A unit test was also added to test_config to test saving of config files. The regression was discovered when app_voicemail's passwordlocation=spooldir feature stopped working. ASTERISK-25917 #close Reported-by: Jonathan Rose Change-Id: Ic4dbe58c277a47b674679e49daed5fc6de349f80	9 years ago
Richard Mudgett	ebf0724a83	test_message.c: Wait longer in case dialplan also processes the test message. ... Bumped the wait from 1 second to 5 seconds. The test message was hitting my default call handler and failing the test because it took longer. Change-Id: I3a03737f25e92983de00548fcc7bbc50dd7544ba	9 years ago
Richard Mudgett	5e388d4188	bridge_softmix.c: Fix crash if channel fails to join mixing tech. ... softmix_bridge_join() failed because of an allocation failure. To address this, the softmix bridge technology now checks if the channel failed to join softmix successfully. In addition, the bridge now begins the process of kicking the channel out of the bridge so we don't have channels partially in the bridge for very long. * Fix the test_channel_feature_hooks.c unit tests. The test channel must have a valid codec to join the simple_bridge technology. This patch makes joining a bridge more strict by not allowing partially joined channels to remain in the bridge. Change-Id: I97e2ade6a2bcd1214f24fb839fda948825b61a2b	9 years ago
George Joseph	9740277713	res_pjsip: Add serialized scheduler (res_pjsip/pjsip_scheduler.c) ... There are several places that do scheduled tasks or periodic housecleaning, each with its own implementation: * res_pjsip_keepalive has a thread that sends keepalives. * pjsip_distributor has a thread that cleans up expired unidentified requests. * res_pjsip_registrar_expire has a thread that cleans up expired contacts. * res_pjsip_pubsub uses ast_sched directly and then calls ast_sip_push_task. * res_pjsip_sdp_rtp also uses ast_sched to send keepalives. There are also places where we should be doing scheduled work but aren't. A good example are the places we have sorcery observers to start registration or qualify. These don't work when changes are made to a backend database without a pjsip reload. We need to check periodically. As a first step to solving these issues, a new ast_sip_sched facility has been created. ast_sip_sched wraps ast_sched but only uses ast_sched as a scheduled queue. When a task is ready to run, ast_sip_task_pusk is called for it. This ensures that the task is executed in a PJLIB registered thread and doesn't hold up the ast_sched thread so it can immediately continue processing the queue. The serializer used by ast_sip_sched is one of your choosing or a random one from the res_pjsip pool if you don't choose one. Another feature is the ability to automatically clean up the task_data when the task expires (if ever). If it's an ao2 object, it will be dereferenced, if it's a malloc'd object it will be freed. This is selectable when the task is scheduled. Even if you choose to not auto dereference an ao2 task data object, the scheduler itself maintains a reference to it while the task is under it's control. This prevents the data from disappearing out from under the task. There are two scheduling models. AST_SIP_SCHED_TASK_PERIODIC specifies that the invocations of the task occur at the specific interval. That is, every "interval" milliseconds, regardless of how long the task takes. If the task takes longer than the interval, it will be scheduled at the next available multiple of interval. For exmaple: If the task has an interval of 60 secs and the task takes 70 secs (it better not), the next invocation will happen at 120 seconds. AST_SIP_SCHED_TASK_DELAY specifies that the next invocation of the task should start "interval" milliseconds after the current invocation has finished. Also, the same ast_sched facility for fixed or variable intervals exists. The task's return code in conjunction with the AST_SIP_SCHED_TASK_FIXED or AST_SIP_SCHED_TASK_VARIABLE flags controls the next invocation start time. One res_pjsip.h housekeeping change was made. The pjsip header files were added to the top. There have been a few cases lately where I've needed res_pjsip.h just for ast_sip calls and had compiles fail spectacularly because I didn't add the pjsip header files to my source even though I never referenced any pjsip calls. Finally, a few new convenience APIs were added to astobj2 to make things a little easier in the scheduler. ao2_ref_and_lock() calls ao2_ref() and ao2_lock() in one go. ao2_unlock_and_unref() does the reverse. A few macros were also copied from res_phoneprov because I got tired of having to duplicate the same hash, sort and compare functions over and over again. The AO2_STRING_FIELD_(HASH|SORT|CMP)_FN macros will insert functions suitable for aor_container_alloc into your source. This facility can be used immediately for the situations where we already have a thread that wakes up periodically or do some scheduled work. For the registration and qualify issues, additional sorcery and schema changes would need to be made so that we can easily detect changed objects on a periodic basis without having to pull the entire database back to check. I'm thinking of a last-updated timestamp on the rows but more on this later. Change-Id: I7af6ad2b2d896ea68e478aa1ae201d6dd016ba1c	9 years ago
George Joseph	772ff3048f	lock: Add named lock capability ... Locking some objects like sorcery objects can be tricky because the underlying ao2 object may not be the same for all callers. For instance, two threads that call ast_sorcery_retrieve_by_id on the same aor name might actually get 2 different ao2 objects if the underlying wizard had to rehydrate the aor from a database. Locking one ao2 object doesn't have any effect on the other even if those objects had locks in the first place. Named locks allow access control by keyspace and key strings. Now an "aor" named "1000" can be locked and any other thread attempting to lock "aor" "1000" will wait regardless of whether the underlying ao2 object is the same or not. Mutex and rwlocks are supported. This capability will initially be used to lock an aor when multiple threads may be attempting to prune expired contacts from it. Change-Id: If258c0b7f92b02d07243ce70e535821a1ea7fb45	9 years ago
George Joseph	f6f4cf459f	stringfields: Refactor to allow fields to be added to the end of structures ... String fields are great, except that you can't add new ones without breaking ABI compatibility because it shifts down everything else in the structure. The only alternative is to add your own char * field to the end of the structure and manage the memory yourself which isn't ideal, especially since you then can't use the OPT_STRINGFIELD_T type. Background: The reason string fields had to be declared inside the AST_DECLARE_STRING_FIELDS block was to facilitate iteration over all declared fields for initialization, compare and copy. Since AST_DECLARE_STRING_FIELDS declared the pool, then the fields, then the manager, you could use the offsets of the pool and manager and iterate over the sequential addresses in between to access the fields. The actual pool, field allocation and field set operations don't actually care where the field is. It's just iteration over the fields that was the problem. Solution: Extended String Fields An extended string field is one that is declared outside the AST_DECLARE_STRING_FIELDS block but still (anywhere) inside the parent structure. Other than using AST_STRING_FIELD_EXTENDED instead of AST_STRING_FIELD, it looks the same as other string fields. It's storage comes from the pool and it participates in string field compare and copy operations peformed on the parent structure. It's also a valid target for the OPT_STRINGFIELD_T aco option type. Implementation: To keep track of the extended fields and make sure that ABI isn't broken, the existing embedded_pool pointer in the manager structure was repurposed to be a pointer to a separate header structure that contains the embedded_pool pointer plus a vector of fields. The length of the manager structure didn't change and the embedded_pool pointer isn't used in the macros, only the stringfields C code. A side benefit of this is that changing the header structure in the future won't break ABI. ast_string_fields_init initializes the normal string fields and appends them to the vector, and subsequent calls to ast_string_field_init_extended initialize and append the extended fields. Cleanup, ast_string_fields_cmp, and ast_string_fields_copy can now work on the vector instead of sequentially traversing the addresses between the pool and manager. The total size of a structure using string fields didn't change, whether using extended fields or not, nor have the offsets of any structure members, either inside the original block or outside. Adding an extended field to the end of a structure is the same as adding a char *. Details: The stringfield C code was pulled out from utils.c and into stringfields.c. It just made sense. Additional work was done in ast_string_field_init and ast_calloc_with_stringfields to handle the allocation of the new header structure and the vector, and the associated cleanup. In the process some additional NULL pointer checking was added. A lot of work was done in stringfields.h since the logic for compare and copy is there. Documentation was added as well as somne additional NULL checking. The ability to call ast_calloc_with_stringfields with a number of structures greater than 1 never really worked. Well, the calloc worked but there was no way to access the additional structures or clean them up. It was agreed that there was no use case for requesting more than 1 structure so an ast_assert was added to prevent it and the iteration code removed. Testing: The stringfield unit tests were updated to test both normal and extended fields. Tests for ast_string_field_ptr_set_by_fields and ast_calloc_with_stringfields were also added. As an ABI test, 13 was compiled from git and the res_pjsip_* modules, except res_pjsip itself, saved off. The patch was then added and a full compile and install was performed. Then the older res_pjsip_* moduled were copied over the installed versions so res_pjsip was new and the rest were old. No issues. contact->aor, which is a char * at the end of contact, was then changed to an extended string field and a recompile and reinstall was performed, again leaving stock versions of the the res_pjsip_* modules. Again, no issues with the res_pjsip_* modules using the old stringfield implementation and with contact->aor as a char *, and res_pjsip itself using the new stringfield implementation and contact->aor being an extended string field. Finally, several existing string fields were converted to extended string fields to test OPT_STRINGFIELD_T. Again, no issues. Change-Id: I235db338c5b178f5a13b7946afbaa5d4a0f91d61	9 years ago
George Joseph	5aa5c49413	sorcery/res_pjsip: Refactor for realtime performance ... There were a number of places in the res_pjsip stack that were getting all endpoints or all aors, and then filtering them locally. A good example is pjsip_options which, on startup, retrieves all endpoints, then the aors for those endpoints, then tests the aors to see if the qualify_frequency is > 0. One issue was that it never did anything with the endpoints other than retrieve the aors so we probably could have skipped a step and just retrieved all aors. But nevermind. This worked reasonably well with local config files but with a realtime backend and thousands of objects, this was a nightmare. The issue really boiled down to the fact that while realtime supports predicates that are passed to the database engine, the non-realtime sorcery backends didn't. They do now. The realtime engines have a scheme for doing simple comparisons. They take in an ast_variable (or list) for matching, and the name of each variable can contain an operator. For instance, a name of "qualify_frequency >" and a value of "0" would create a SQL predicate that looks like "where qualify_frequency > '0'". If there's no operator after the name, the engines add an '=' so a simple name of "qualify_frequency" and a value of "10" would return exact matches. The non-realtime backends decide whether to include an object in a result set by calling ast_sorcery_changeset_create on every object in the internal container. However, ast_sorcery_changeset_create only does exact string matches though so a name of "qualify_frequency >" and a value of "0" returns nothing because the literal "qualify_frequency >" doesn't match any name in the objset set. So, the real task was to create a generic string matcher that can take a left value, operator and a right value and perform the match. To that end, strings.c has a new ast_strings_match(left, operator, right) function. Left and right are the strings to operate on and the operator can be a string containing any of the following: = (or NULL or ""), !=, >, >=, <, <=, like or regex. If the operator is like or regex, the right string should be a %-pattern or a regex expression. If both left and right can be converted to float, then a numeric comparison is performed, otherwise a string comparison is performed. To use this new function on ast_variables, 2 new functions were added to config.c. One that compares 2 ast_variables, and one that compares 2 ast_variable lists. The former is useful when you want to compare 2 ast_variables that happen to be in a list but don't want to traverse the list. The latter will traverse the right list and return true if all the variables in it match the left list. Now, the backends' fields_cmp functions call ast_variable_lists_match instead of ast_sorcery_changeset_create and they can now process the same syntax as the realtime engines. The realtime backend just passes the variable list unaltered to the engine. The only gotcha is that there's no common realtime engine support for regex so that's been noted in the api docs for ast_sorcery_retrieve_by_fields. Only one more change to sorcery was done... A new config flag "allow_unqualified_fetch" was added to reg_sorcery_realtime. "no": ignore fetches if no predicate fields were supplied. "error": same as no but emit an error. (good for testing) "yes": allow (the default); "warn": allow but emit a warning. (good for testing) Now on to res_pjsip... pjsip_options was modified to retrieve aors with qualify_frequency > 0 rather than all endpoints then all aors. Not only was this a big improvement in realtime retrieval but even for config files there's an improvement because we're not going through endpoints anymore. res_pjsip_mwi was modified to retieve only endpoints with something in the mailboxes field instead of all endpoints then testing mailboxes. res_pjsip_registrar_expire was completely refactored. It was retrieving all contacts then setting up scheduler entries to check for expiration. Now, it's a single thread (like keepalive) that periodically retrieves only contacts whose expiration time is < now and deletes them. A new contact_expiration_check_interval was added to global with a default of 30 seconds. Ross Beer reports that with this patch, his Asterisk startup time dropped from around an hour to under 30 seconds. There are still objects that can't be filtered at the database like identifies, transports, and registrations. These are not going to be anywhere near as numerous as endpoints, aors, auths, contacts however. Back to allow_unqualified_fetch. If this is set to yes and you have a very large number of objects in the database, the pjsip CLI commands will attempt to retrive ALL of them if not qualified with a LIKE. Worse, if you type "pjsip show endpoint <tab>" guess what's going to happen? :) Having a cache helps but all the objects will have to be retrieved at least once to fill the cache. Setting allow_unqualified_fetch=no prevents the mass retrieve and should be used on endpoints, auths, aors, and contacts. It should NOT be used for identifies, registrations and transports since these MUST be retrieved in bulk. Example sorcery.conf: [res_pjsip] endpoint=config,pjsip.conf,criteria=type=endpoint endpoint=realtime,ps_endpoints,allow_unqualified_fetch=error ASTERISK-25826 #close Reported-by: Ross Beer Tested-by: Ross Beer Change-Id: Id2691e447db90892890036e663aaf907b2dc1c67	9 years ago
Richard Mudgett	b2d2906445	sched.c: Ensure oldest expiring entry runs first. ... This patch is part of a series to resolve deadlocks in chan_sip.c. * Updated sched unit test to check new behavior. ASTERISK-25023 Change-Id: Ib69437327b3cda5e14c4238d9ff91b2531b34ef3	9 years ago
Mark Michelson	f1f79812c1	Fix failing threadpool_auto_increment test. ... The threadpool_auto_increment test fails infrequently for a couple of reasons * The threadpool listener was notified of fewer tasks being pushed than were actually pushed * The "was_empty" flag was set to an unexpected value. The problem is that the test pushes three tasks into the threadpool. Test expects the threadpool to essentially gather those three tasks, and then distribute those to the threadpool threads. It also expects that as the tasks are pushed in, the threadpool listener is alerted immediately that the tasks have been pushed. In reality, a task can be distributed to the threadpool threads quicker than expected, meaning that the threadpool has already emptied by the time each subsequent task is pushed. In addition, the internal threadpool queue can be delayed so that the threadpool listener is not alerted that a task has been pushed even after the task has been executed. From the test's point of view, there's no way to be able to predict exactly the order that task execution/listener notifications will occur, and there is no way to know which listener notifications will indicate that the threadpool was previously empty. For this reason, the test has been updated to only check the things it can check. It ensures that all tasks get executed, that the threads go idle after the tasks are executed, and that the listener is told the proper number of tasks that were pushed. Change-Id: I7673120d74adad64ae6894594a606e102d9a1f2c	9 years ago
Joshua Colp	a7c8d4cd6b	tests/test_sorcery_memory_cache_thrash: Improve termination process. ... When terminating the threads thrashing a sorcery memory cache each would be told to stop and then we would wait on them. During at least one thrashing test this was problematic due to the specific usage pattern in use. It would take some time for termination of the thread to occur. This would occur due to contention between the threads retrieving and the threads updating the cache. As the retrieving threads are given priority it may be some time before the updating threads are able to proceed. This change makes it so all threads are told to stop and then each are joined to ensure they stop. This way all the threads should stop at around the same time instead of waiting for one to stop, the next to stop, then the next, and so on. As a result of this the execution time for each thrash test is much closer to their expected value than previously seen as well. Change-Id: I04a53470b0ea4170b8819180b0bd7475f3642827	9 years ago