25 Commits (87263b47b51bf727e04f352f988acb508924e423)

Author	SHA1	Message	Date
Matthew Jordan	f1b4969a33	tests/test_stasis: Resolve compilation issues from Asterisk 12 merge ... When merging the changes up stream in r428687, I missed the fact that the signature for stasis_message_type_create was changed. This patch fixes the compilation issues introduced by that merge. git-svn-id: https://origsvn.digium.com/svn/asterisk/branches/13@428815 65c4cc65-6c06-0410-ace0-fbb531ad65f3	11 years ago
Matthew Jordan	d79c68d3fb	main/stasis: Allow subscriptions to use a threadpool for message delivery ... Prior to this patch, all Stasis subscriptions would receive a dedicated thread for servicing published messages. In contrast, prior to r400178 (see review https://reviewboard.asterisk.org/r/2881/), the subscriptions shared a thread pool. It was discovered during some initial work on Stasis that, for a low subscription count with high message throughput, the threadpool was not as performant as simply having a dedicated thread per subscriber. For situations where a subscriber receives a substantial number of messages and is always present, the model of having a dedicated thread per subscriber makes sense. While we still have plenty of subscriptions that would follow this model, e.g., AMI, CDRs, CEL, etc., there are plenty that also fall into the following two categories: * Large number of subscriptions, specifically those tied to endpoints/peers. * Low number of messages. Some subscriptions exist specifically to coordinate a single message - the subscription is created, a message is published, the delivery is synchronized, and the subscription is destroyed. In both of the latter two cases, creating a dedicated thread is wasteful (and in the case of a large number of peers/endpoints, harmful). In those cases, having shared delivery threads is far more performant. This patch adds the ability of a subscriber to Stasis to choose whether or not their messages are dispatched on a dedicated thread or on a threadpool. The threadpool is configurable through stasis.conf. Review: https://reviewboard.asterisk.org/r/4193 ASTERISK-24533 #close Reported by: xrobau Tested by: xrobau ........ Merged revisions 428681 from http://svn.asterisk.org/svn/asterisk/branches/12 git-svn-id: https://origsvn.digium.com/svn/asterisk/branches/13@428687 65c4cc65-6c06-0410-ace0-fbb531ad65f3	11 years ago
Kinsey Moore	f1036f40dc	Stasis: Allow message types to be blocked ... This introduces stasis.conf and a mechanism to prevent certain message types from being published. Internally, this works by preventing the chosen message types from being created which ensures that those message types can never be published. This patch also adjusts message publishers such that message payloads are not created if the related message type is not available. ASTERISK-23943 #close Review: https://reviewboard.asterisk.org/r/3823/ git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@420124 65c4cc65-6c06-0410-ace0-fbb531ad65f3	11 years ago
Richard Mudgett	4ad1245cb5	stasis cache: Enhance to keep track of an item from different entities. ... A stasis cache entry now contains more than a single message/snapshot. It contains messages/snapshots for the local entity as well as any remote entities that post to the cached item. In addition callbacks can be supplied when the cache is created to compute and post the aggregate message/snapshot representing all entities stored in the cache entry. * All stasis messages now have an eid to indicate what entity posted it. * The stasis cache enhancements allow device state to cache and aggregate the device states from local and remote entities in a single operation. The cached aggregate device state is available immediately after it is posted to the stasis bus. This improves performance by eliminating a cache dump and associated ao2 container traversals to calculate the aggregate state. (closes issue ASTERISK-23204) Reported by: Mark Michelson Review: https://reviewboard.asterisk.org/r/3281/ ........ Merged revisions 410184 from http://svn.asterisk.org/svn/asterisk/branches/12 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@410185 65c4cc65-6c06-0410-ace0-fbb531ad65f3	11 years ago
Richard Mudgett	f8dd49ab4d	test_stasis.c: Misc cleanups. ... * Make the consumer ao2 object use the ao2 lock instead of a redundant lock in the struct for ast_cond_wait(). * Fixed some curly brace placements. * Fixed use of malloc(0). malloc(0) has variant behavior. It is up to the implementation to determine if it returns NULL or a valid pointer that can be later passed to free(). ........ Merged revisions 408983 from http://svn.asterisk.org/svn/asterisk/branches/12 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@408984 65c4cc65-6c06-0410-ace0-fbb531ad65f3	11 years ago
Matthew Jordan	f8aaf585a3	stasis: Add methods to allow for synchronous publishing to subscriber ... This patch adds an API call to Stasis that allows a publisher to publish a stasis message that will not return until a specific subscriber handles the message. Since a subscriber can have their own forwarding topic which orders messages from many topics, this allows a publisher who knows of that subscriber to synchronize to that subscriber regardless of the forwarding relationships between topics. This is of particular use for dialplan applications that need to synchronize on a particular subscriber's handling of a message. (issue ASTERISK-22884) Reported by: Matt Jordan Review: https://reviewboard.asterisk.org/r/3099/ ........ Merged revisions 405311 from http://svn.asterisk.org/svn/asterisk/branches/12 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@405313 65c4cc65-6c06-0410-ace0-fbb531ad65f3	11 years ago
Richard Mudgett	c2caaecc45	test_stasis.c: Fix ref leak in normal execution path. ... ........ Merged revisions 404764 from http://svn.asterisk.org/svn/asterisk/branches/12 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@404783 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
Richard Mudgett	72c282cc66	ao2_iterator: Mini-audit of the ao2_iterator loops in the new code files. ... * Fixed several places where ao2_iterator_destroy() was not called. * Fixed several iterator loop object variable reference problems. * Fixed res_parking AMI actions returning non-zero. Only the AMI logoff action can return non-zero. Review: https://reviewboard.asterisk.org/r/3087/ ........ Merged revisions 404434 from http://svn.asterisk.org/svn/asterisk/branches/12 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@404436 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
Kinsey Moore	2c90d80b8f	Make sure unit tests compile ... This fixes the unit tests that were broken by r403069 and several functions requiring a new parameter for sanitization of JSON messages generated from object snapshots. ........ Merged revisions 403094 from http://svn.asterisk.org/svn/asterisk/branches/12 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@403095 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
David M. Lee	2de42c2a25	Multiple revisions 399887,400138,400178,400180-400181 ... ........ r399887 | dlee | 2013-09-26 10:41:47 -0500 (Thu, 26 Sep 2013) | 1 line Minor performance bump by not allocate manager variable struct if we don't need it ........ r400138 | dlee | 2013-09-30 10:24:00 -0500 (Mon, 30 Sep 2013) | 23 lines Stasis performance improvements This patch addresses several performance problems that were found in the initial performance testing of Asterisk 12. The Stasis dispatch object was allocated as an AO2 object, even though it has a very confined lifecycle. This was replaced with a straight ast_malloc(). The Stasis message router was spending an inordinate amount of time searching hash tables. In this case, most of our routers had 6 or fewer routes in them to begin with. This was replaced with an array that's searched linearly for the route. We more heavily rely on AO2 objects in Asterisk 12, and the memset() in ao2_ref() actually became noticeable on the profile. This was #ifdef'ed to only run when AO2_DEBUG was enabled. After being misled by an erroneous comment in taskprocessor.c during profiling, the wrong comment was removed. Review: https://reviewboard.asterisk.org/r/2873/ ........ r400178 | dlee | 2013-09-30 13:26:27 -0500 (Mon, 30 Sep 2013) | 24 lines Taskprocessor optimization; switch Stasis to use taskprocessors This patch optimizes taskprocessor to use a semaphore for signaling, which the OS can do a better job at managing contention and waiting that we can with a mutex and condition. The taskprocessor execution was also slightly optimized to reduce the number of locks taken. The only observable difference in the taskprocessor implementation is that when the final reference to the taskprocessor goes away, it will execute all tasks to completion instead of discarding the unexecuted tasks. For systems where unnamed semaphores are not supported, a really simple semaphore implementation is provided. (Which gives identical performance as the original taskprocessor implementation). The way we ended up implementing Stasis caused the threadpool to be a burden instead of a boost to performance. This was switched to just use taskprocessors directly for subscriptions. Review: https://reviewboard.asterisk.org/r/2881/ ........ r400180 | dlee | 2013-09-30 13:39:34 -0500 (Mon, 30 Sep 2013) | 28 lines Optimize how Stasis forwards are dispatched This patch optimizes how forwards are dispatched in Stasis. Originally, forwards were dispatched as subscriptions that are invoked on the publishing thread. This did not account for the vast number of forwards we would end up having in the system, and the amount of work it would take to walk though the forward subscriptions. This patch modifies Stasis so that rather than walking the tree of forwards on every dispatch, when forwards and subscriptions are changed, the subscriber list for every topic in the tree is changed. This has a couple of benefits. First, this reduces the workload of dispatching messages. It also reduces contention when dispatching to different topics that happen to forward to the same aggregation topic (as happens with all of the channel, bridge and endpoint topics). Since forwards are no longer subscriptions, the bulk of this patch is simply changing stasis_subscription objects to stasis_forward objects (which, admittedly, I should have done in the first place.) Since this required me to yet again put in a growing array, I finally abstracted that out into a set of ast_vector macros in asterisk/vector.h. Review: https://reviewboard.asterisk.org/r/2883/ ........ r400181 | dlee | 2013-09-30 13:48:57 -0500 (Mon, 30 Sep 2013) | 28 lines Remove dispatch object allocation from Stasis publishing While looking for areas for performance improvement, I realized that an unused feature in Stasis was negatively impacting performance. When a message is sent to a subscriber, a dispatch object is allocated for the dispatch, containing the topic the message was published to, the subscriber the message is being sent to, and the message itself. The topic is actually unused by any subscriber in Asterisk today. And the subscriber is associated with the taskprocessor the message is being dispatched to. First, this patch removes the unused topic parameter from Stasis subscription callbacks. Second, this patch introduces the concept of taskprocessor local data, data that may be set on a taskprocessor and provided along with the data pointer when a task is pushed using the ast_taskprocessor_push_local() call. This allows the task to have both data specific to that taskprocessor, in addition to data specific to that invocation. With those two changes, the dispatch object can be removed completely, and the message is simply refcounted and sent directly to the taskprocessor. Review: https://reviewboard.asterisk.org/r/2884/ ........ Merged revisions 399887,400138,400178,400180-400181 from http://svn.asterisk.org/svn/asterisk/branches/12 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@400186 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
David M. Lee	f29d969a79	Stasis: address refcount races; implementation comments ... Change r395954 reordered some stasis object destruction, which should have been fine. Unfortunately, it caused some hard to reproduce issues related to objects being accessed after they had been destroyed. The patch in r396329 fixed the destruction order problem; this patch addresses the underlying issue. A few other stasis-related fixes were also added. * Add ref-bumps around areas where objects may get transitively destroyed. (For example, where we lock a topic, unref a subscription, which unrefs the topic, which explodes the topic when we try to unlock it.) * Wrote an extensive doxygen page about Stasis implementation, relationships between objects, lifecycles of objects, how the refcounting works, etc. Many other comments were added, corrected, or cleaned up. * Added an assert to the topic dtor to catch extra ref decrements. * Fixed type used after destruction errors for graceful shutdown in stasis_channels.c. * I added two unit tests in an attempt to catch destruction order issues. Since the underlying cause is a race condition, though, the tests rarely failed even when the code was wrong. * Fixed a leak in stasis_cache_pattern.c. (closes issue ASTERISK-22243) Review: https://reviewboard.asterisk.org/r/2746/ git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@396842 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
Richard Mudgett	1d57078837	Fix stasis/core unit test. Should have had the CR/LF. ... git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@396480 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
Richard Mudgett	154f45dd02	Add missing CR/LF to FakeMI stasis test AMI event. ... git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@396463 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
David M. Lee	b97d318b7b	Tweak caching topics to fix CEL tests ... The Stasis changes in r395954 had an unanticipated side effect: messages published directly to an _all topic does not get forwarded to the corresponding caching topic. This patch fixes that by changing how caching topics forward messages, and how the caching pattern forwards are setup. For the caching pattern, the all_topic is forwarded to the all_topic_cached. This forwards messages published directly to the all_topic to all_topic_cached. In order to avoid duplicate messages on all_topic_cached, caching topics were changed to no longer forward uncached messages. Subscribers to an individual caching topic should only expect to receive cache updates, and subscription change messages. Since individual caching topics are new, this shouldn't be a problem. There are a few minor changes to the pre-cache split behavior. * For topics changed to use the caching pattern, the all_topic_cached will forward snapshots in addition to cache updates. Since subscribers by design ignore unexpected messages, this should be fine. * Caching topics that don't use the caching pattern no longer forward non-cache updates. This makes no difference for the current caching topics. * mwi_topic_cached, channel_by_name_topic and presence_state_topic_cached have no subscribers * device_state_topic_cached's only subscriber only processes cache udpates (issue ASTERISK-22243) Review: https://reviewboard.asterisk.org/r/2738 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@396329 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
David M. Lee	e1b959ccbb	Split caching out from the stasis_caching_topic. ... In working with res_stasis, I discovered a significant limitation to the current structure of stasis_caching_topics: you cannot subscribe to cache updates for a single channel/bridge/endpoint/etc. To address this, this patch splits the cache away from the stasis_caching_topic, making it a first class object. The stasis_cache object is shared amongst individual stasis_caching_topics that are created per channel/endpoint/etc. These are still forwarded to global whatever_all_cached topics, so their use from most of the code does not change. In making these changes, I noticed that we frequently used a similar pattern for bridges, endpoints and channels: single_topic ----------------> all_topic ^ | single_topic_cached ----+----> all_topic_cached | +----> cache This pattern was extracted as the 'Stasis Caching Pattern', defined in stasis_caching_pattern.h. This avoids a lot of duplicate code between the different domain objects. Since the cache is now disassociated from its upstream caching topics, this also necessitated a change to how the 'guaranteed' flag worked for retrieving from a cache. The code for handling the caching guarantee was extracted into a 'stasis_topic_wait' function, which works for any stasis_topic. (closes issue ASTERISK-22002) Review: https://reviewboard.asterisk.org/r/2672/ git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@395954 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
David M. Lee	fec667646f	Fix bridge/channel AMI event ordering issues ... The stasis_cache_update messages are somewhat cumbersome to handle with the stasis_message_router. Since all updates have the same message type, they are normally handled with the same route. Since caching itself is a first class component of stasis-core, it makes sense for the router to handle the cache update messages itself. This patch adds stasis_message_router_add_cache_update() and stasis_message_router_remove_cache_update() to handle the routing of stasis_cache_update messages. This patch also corrects an issue with manager_{bridging,channels}.c, where events might be reordered. The reordering occurs because the components use different message routers, which they needed because they both needed to route cache update messages. They now both use manager's router, and add cache routes for just the cache updates they are interested in. (closes issue ASTERISK-22038) Review: https://reviewboard.asterisk.org/r/2677/ git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@395118 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
David M. Lee	dbdb2b1b3a	Add vtable and methods for to_json and to_ami for Stasis messages ... When a Stasis message type is defined in a loadable module, handling those messages for AMI and res_stasis events can be cumbersome. This patch adds a vtable to stasis_message_type, with to_ami and to_json virtual functions. These allow messages to be handled abstractly without putting module-specific code in core. As an example, the VarSet AMI event was refactored to use the to_ami virtual function. (closes issue ASTERISK-21817) Review: https://reviewboard.asterisk.org/r/2579/ git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@391403 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
Kinsey Moore	759a7e4a30	Rework stasis cache clear events ... Stasis cache clear message payloads now consist of a stasis_message representative of the message to be cleared from the cache. This allows multiple parallel caches to coexist and be cleared properly by the same cache clear message even when keyed on different fields. This change fixes a bug where multiple cache clears could be posted for channels. The cache clear is now produced in the destructor instead of ast_hangup. Additionally, dummy channels are no longer capable of producing channel snapshots. Review: https://reviewboard.asterisk.org/r/2596 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@390830 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
David M. Lee	1c21b8575b	This patch adds a RESTful HTTP interface to Asterisk. ... The API itself is documented using Swagger, a lightweight mechanism for documenting RESTful API's using JSON. This allows us to use swagger-ui to provide executable documentation for the API, generate client bindings in different languages, and generate a lot of the boilerplate code for implementing the RESTful bindings. The API docs live in the rest-api/ directory. The RESTful bindings are generated from the Swagger API docs using a set of Mustache templates. The code generator is written in Python, and uses Pystache. Pystache has no dependencies, and be installed easily using pip. Code generation code lives in rest-api-templates/. The generated code reduces a lot of boilerplate when it comes to handling HTTP requests. It also helps us have greater consistency in the REST API. (closes issue ASTERISK-20891) Review: https://reviewboard.asterisk.org/r/2376/ git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@386232 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
David M. Lee	b23e8e1950	stasis: Fixed message ordering issues when forwarding ... This patch fixes an issue of message ordering that occurs when multiple topics are forwarded to an aggregator topic (such as ast_channel_topic_all()). It is (very reasonably) expected that the rules governing message dispatch order still apply, so long as the messages start from the same thread, and are received by the same subscription. Because the existing code had an additional layer of dispatching via the Stasis thread pool for forwards, those promises couldn't be kept. Forwarding subscriptions no longer have their own mailbox, and now dispatch directly from the forwarding topic's stasis_publish() call. This means that the topic's lock is held for the duration of not only a message's dispatch, but the dispatch of all the forwards. This shouldn't be a problem right now, but if an aggregator topic had many subscribers, it could become a problem. But I figure we can write more clever code when the time comes, if necessary. Review: https://reviewboard.asterisk.org/r/2419/ git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@384413 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
Kinsey Moore	71206544a7	Break the world. Stasis message type accessors should now all be named correctly. ... git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@384261 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
David M. Lee	49e3489cac	A simplistic router for stasis_message's. ... Often times, when subscribing to a topic, one wants to handle different message types differently. While one could cascade if/else statements through the subscription handler, it is much cleaner to specify a different callback for each message type. The stasis_message_router is here to help! A stasis_message_router is constructed for a particular stasis_topic, which is subscribes to. Call stasis_message_router_unsubscribe() to cancel that subscription. Once constructed, routes can be added using stasis_message_router_add() (or stasis_message_router_set_default() for any messages not handled by other routes). There may be only one route per stasis_message_type. The route's callback is invoked just as if it were a callback for a subscription; but it only gets called for messages of the specified type. (issue ASTERISK-20887) Review: https://reviewboard.asterisk.org/r/2390/ git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@383242 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
Kinsey Moore	ccb5526508	Take advantage of the fact that stasis_unsubscribe now returns NULL ... git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@383169 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
Kinsey Moore	c6b06e40dc	Add message dump capability to stasis cache layer ... The cache dump mechanism allows the developer to retreive multiple items of a given type (or of all types) from the cache residing in a stasis caching topic in addition to the existing single-item cache retreival mechanism. This also adds to the caching unit tests to ensure that the new cache dump mechanism is functioning properly. Review: https://reviewboard.asterisk.org/r/2367/ (issue ASTERISK-21097) git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@382705 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
David M. Lee	4edd8be35c	This patch adds a new message bus API to Asterisk. ... For the initial use of this bus, I took some work kmoore did creating channel snapshots. So rather than create AMI events directly in the channel code, this patch generates Stasis events, which manager.c uses to then publish the AMI event. This message bus provides a generic publish/subscribe mechanism within Asterisk. This message bus is: - Loosely coupled; new message types can be added in seperate modules. - Easy to use; publishing and subscribing are straightforward operations. In addition to basic publish/subscribe, the patch also provides mechanisms for message forwarding, and for message caching. (issue ASTERISK-20887) (closes issue ASTERISK-20959) Review: https://reviewboard.asterisk.org/r/2339/ git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@382685 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago