178 Commits (b041888807ab89a0624bd7f9ba75e21ec53861cf)

Author	SHA1	Message	Date
Kirill Solomko	b041888807	MT#60656 add max_age password validation support ... * UI: password are now validated against $c->config->{security}{password}{web_max_age_days} (unless it's 0) and if the password is expired the user is redirected automatically to /changepassword page, and after successful password change back to the original page. * API: if password is expired all API requests will be returning 403 Forbidden "Password expired", except PUT/PATCH to /api/admins or /api/subscribers with the new password in place. * successful login on the UI now redirects to /dashboard instead of / (to prevent unintended redirect to v2) Change-Id: I075f8e17cc9b0658d6b3b3d526ca5b379d050ce4	9 months ago
Kirill Solomko	65ae07a97b	MT#60601 obfuscate password/webpassword in API valdiation errors ... * plain text passwords are obfuscated in validation errors returned by API Change-Id: Iff989696ce09faff34692eaa129aee6981340a97	9 months ago
Kirill Solomko	9d021be65a	MT#59979 add license control ... * UI and API parts are now under license control * new Util::License::get_license($c, $name) - fetches license status by name (1 if enabled, and also if /proc/ngcp/check if 'ok') * add Catalyst::Plugins::NGCP::License with license($name) to fetch valid license by name from anywhere using $c->license('pbx') or from the templates using c.license('pbx'). It internally uses Util::License::get_license($c, $name) * License::get_license_status($c) now requires $c as first argument as well logs license status check errors. * new ActionRoles::License that enables usage of :Does(License) RequiresLicense('pbx') LicenseDetachTo('/denied_page') in the Controller chains * Add license control for UI elements and return 403 Forbidden if a resource is covered by licenses and the license is not active * Hide UI elements if a license is not active * API/Entities/Entities new $c->set_config key: - per endpoint: $c->set_config({ required_licenses => [qw/pbx device_provisioning/] } - or per method: $c->set_config({ required_licenses => { POST => [qw/pbx device_provisioning/] } } } * In case if an API endpoint does not have a license: 403 Forbidden "Invalid license" reply is returned. * Add license based restrictions to API endpoints * /api documentation: - completely hide endpoints that do not have an active license - hide only methods that does not have an active license Change-Id: Iba45fc5068b02306a617fed7b5405f2210574b61	9 months ago
Kirill Solomko	524a264850	MT#60498 add search by not_null, change null param ... * add search by not_null for fields that ends with _id and integers, that adds "IS NOT NULL" to the SQL search query by the field. * null and not null values are expected now as $null and $not_null to avoid conflicts when user searches by null strings that may be a valid case. Change-Id: I8e8b8c9060e985dfe2b94cbfcca1587f05477fe9	9 months ago
Kirill Solomko	18597efb87	MT#60378 accept null same way as NULL for search ... * search by value null is internally translated to undef to say search for records without a value, same way as it already works for NULL value, just null is more expected and easier to use by clients as it's how its represented in the resource Change-Id: Ia8a75bad95a34dd8167162d0f09a1ec7c4056105	10 months ago
Kirill Solomko	6543743992	MT#60162 improve expand functionality for collections ... * in general expand collection now performs only 1 sql query to per expand field to fetch all items by the ids instead of fetching them for every single collection item, that should significantly increase performance in case of large databases and reduce work for the database * introduce $c->stash->{expand_cache} that contains cached data for the expand fields to avoid multiple same calculations * expand_field() and expand_field_data() have been reworked to support expand_cache * new method expand_prepare_collection() is called for all API GET collection methods before preparing resource hal fields, to change the expand logic to only cache the data instead of fetching it from the database * new method expand_collection_fields() that is called in all API GET collection methods after the @embedded data is prepared to finalise the expand collection fields * for expand collection there is only 1 SQL request per expand field that fetches all items -in [ids to expand] and then the expand_collection_fields() uses the cached items_by_ids with O(1) fetch from the cache by id Change-Id: Ie7c6115472878febf0d8c9b4d833f5c23b15c78b	10 months ago
Michael Prokop	c8766cd567	MT#48844 Minor typo fixes ... s/paramaters/parameters/ s/wihout/without/ s/If it mathes/If it matches/ s/ in in / in / s/the the/the/ (note: fixes translation issue which was introduced with commit 2b7a1a33 and fixed in 072e897c for the original, but missed for the german translation) Change-Id: I4186bf3f42b1fac11bb7d7fe801f860d2d59adc1	1 year ago
Kirill Solomko	91f1dcf37e	MT#59478 /api/ root error handling ... * clear errors in Controller::API::Root::end so that they are not interfere with generated ones * API::error(): add $c->stash->{is_api_error_response} = 1 to track it in log_response() * API::log_response(): distinguish and correctly correct errors for cases when is_api_response == 1 and also when there were $c->error('error') calls in the code, to log them correctly in both cases Change-Id: Id922b8219832c1b99815d9d608309035b9b25cff	1 year ago
Kirill Solomko	9d598d4502	MT#59478 improve MSG, LOG log string data storing ... * $c->error array now contains the $message as the first element instead, so that it's possible to obtain all the error data in the code when fetching it from @{$c->error}. the first element is not logged in the error log. * api_response $c->response_body part is now stored in MSG= and possible errors / other log data is now stored in LOG= to: - reduce amount of log lines when an API error response occurs from 2 to 1 - the message part usually contains either HTTP response message (e.g. Internal Server Error) or a validation message string, so it belongs to the MSG= part of the log line, where as the internal log data is more related to the LOG= part - both MSG= and LOG= parts are escaped for GDPR related obfuscation * Utils::Messag::info(): $msg is now also obfuscated if it's detected as a reference (also because logging is moved for the API part to $msg), as well as truncated for possible new-line char and white-spaces. Change-Id: I3b670b2251ec3060037ed6863f18d95975120b8d	1 year ago
Kirill Solomko	205b27a267	MT#59478 API better transaction and error handling ... * Role/Entities*: POST/PUT/PATCH/DELETE methods changes: - support deadlock detection and transaction retry (2 retry attempts at the moment) - improve transaction control, use local $guard instead of saving the ref to $c->stash, as in that case it went out of scope too late and also reported an error message into the log about abnormal $guard out of scope interruption - move all non transaction related code outside of the scope - add error handling when methods such as update_item, and a like do not return the expected data, instead of simply going out of scope and resulting in an uncontrolled reply * Role/API: - rework transaction control: + get_transaction_control() is renamed to start_transaction() to better reflect what it does + complete_transaction() is renamed to commit_transaction() + remove unused %params arg + pass $guard into commit_transaction() instead of having it stored as $c->stash->{transaction_guard) that caused the $guard ref to be destroyed much late than expected (there was also a typo as transaction_quard, which is not relevant anymore with the changes + add check_deadlock() that is invoked when an exception is caught or an $c->errors contain an error, and if the error message represents a transaction error, the transaction block is re-invoked via "goto TX_START" - rework error(): + it now accepts args as following: ($self, $c, $code, $message, @errors) # code -> returned as HTTP code in the reply # message -> returned as HTTP message in the reply # errors -> contain errors for internal logging, last element often contains a DBIx exception + populates all @errors into $c->error so they are available on demend in the code via $c->error or $c->last_error + $c->log->error is not invoked now as the errors become printed in log_response() - log_response() now prints collected errors from $c->error correctly as a separate log line, that is alike to the other api logs so that those can be looked up by the request's tx_id, also all errors are now printed only into api.log * Adjust all $self->error() calls in catch($e) to include $e as the last argument, as well as the duplicate $c->log->error is removed from those ocassions * Remove all $c->log->error() calls as they are replaced with either $self->error() (that logs it correctly into api.log) or $c->error('err') that also adds it correctly into api.log * API::CallForwards: rework to use Entities/EntitiesItem * API::Contracts: rework POST to use Entities * API::PeeringGroups: rework POST to use Entities * API::SubscriberRegistrations: rework POST to use Entities * API::RewriteRuleSets: improve create_item() functionality * Utils/Message: add 'api_retry' log type * $c->session->{api_request_tx_id} is changed to $c->stash->{api_request_tx_id} because sometimes the session ref is different and a different tx_id becomes used Change-Id: I633ce7a8047b1bf00a2f6889003088edf0825dcd	1 year ago
Rene Krenn	868e2657b6	MT#59597 fix remove trailing+leading % entirely for "wildcard" params ... Change-Id: I371e2d41ff27b2801ead391778d9a2bdef914563	1 year ago
Rene Krenn	9ba93e69a5	MT#59597 remove trailing+leading % entirely for "wildcard" param types ... this code is not run by legacy panel, so the check for "/api" request can be removed. Change-Id: I29410fbbf046a8f00443138720b940de0f09f416	1 year ago
Rene Krenn	aa6760228e	MT#59363 retain trailing+leading wildcard for legacy panel only ... all api query param filters that were harmonized to "wildcard" also appended leading+trailing wildcard (this was from the original idea to make adminUI and legacy panel search behave the same). Change-Id: I017473b84f2844d14a11b4ec8b437df5335c24d0	1 year ago
Rene Krenn	c837dc1043	MT#56817 MT#56362 fix query param sorting for regulary chained rs ... Change-Id: I82d3a0ce67ffb61273fc716016885f6431852eff	1 year ago
Rene Krenn	aae0b7e071	MT#56817 MT#56362 fix query param sorting for rs chained the regular way ... Change-Id: I0dab56ae07666eb695d7f30ef633a876ff8258fb	1 year ago
Rene Krenn	a15d00894a	MT#56817 MT#56362 fix errors from nondetermistic query param ordering ... the /api/callists rail supports the "call_id" query parameter to match calls with a callid prefix. this filter also adds an implicity ordering ORDER BY length(call_id) ASC, 'start_time' ASC. the /api/callists api rail also requires the query parameter "susbcriber_id", which renders a (fast) compound SQL query to list matching incoming (query1) and outgoing calls (query2) using UNION ALL (instead of a slow OR query). query1 UNION ALL query2 this is supported by the https://metacpan.org/pod/DBIx::Class::Helper::ResultSet::SetOperations module, which however generates invalid SQL syntax if query1 or query2 contains an ORDER BY. this is exactly what caused the 500 error of the customer who applied both "call_id" and "susbcriber_id" parameter in the request at the same time. ... ORDER BY length(call_id) ASC, `start_time` ASC UNION ALL SELECT `me`.`id`... the error happens randomly, because query parameters are stored in a hash (which by definition has no deterministic order of the entries). when the "call_id" parameter is applied at last, it worked as expected. so the issue can be adressed by strictly ordering the UNION ALL result, and prohibit ORDER BY in query1 and query1. the latter was added already with commit b2dfe28eed, and could be hotfixed. however, the ORDER BY of "call_id" query param is lost, and should be properly handled by forcing strict ordering of how query parameters are applied. the fix will introduce paramater ordering according to their order of definition in the code. Change-Id: I165d341b5c20e9bb750bd1fba88c836b393e80bd	1 year ago
Rene Krenn	308a974059	MT#58762 harmonize restapi wildcard filters ... - all standard LIKE search are migrated - will avoid LIKE unless a pattern (* wildcard) is used as a search term. this encourage db index usage, will be faster - supports wildcard escape sequence \\* - harmonize swagger UI descriptions of filters Change-Id: Iea155871c9be6c284e6970a562d4e6af73fedc4b	1 year ago
Rene Krenn	8e82715eb8	MT#56239 api/callrecordings support filtering multiple metakeys ... filtering an Entity-Attribute-Value model cannot be done as simple conjunctions, but requires either INTERSECT set operation, or joining the same table multiple times. Change-Id: I5ce1ae1ece9406b6610487654f09d768a233b122	1 year ago
Rene Krenn	9c3a549094	MT#56239 api general caller/callee wildcard search support ... various api rails will need to support ?caller= and ?callee= url query parameters. since this involves SQL queries against potentially large database tables, special care is taken with wildcard search to prevent slow queries: - the ?wildcards=true query parameter has to be specified to accept search patterns that contain wildcard symbols, so wildcards are not accepted by default. WARNING: a search string with a leading wildcard will always force a *slow* full db table scan! - the * symbol is used as a wildcard symbol - \ (backslash) is used as escape character to search for a literal '*' Change-Id: Ie6065b0cfa883f7963e1dc8259fffea9a1edfdfe	1 year ago
Kirill Solomko	c58a8b9b1f	MT#57100 avoid where ambiguity for query_types search_eq,search_like ... * uses prefix 'me.' for query types search_eq and search_like in the where condition to avoid ambiguity if the search_rs has joins that with the same column names Change-Id: I90fef80970aa4415480b00bbed2fb9fbee1f1ccc	2 years ago
Kirill Solomko	0555f32585	TT#65101 expand now fills the same key only once ... * the same key is now expanded only once to use the same data ref otherwise expand like contract_id.contact_id,contract_id.billing_profile_id makes it so that 2nd contract_id would overwrite the already existing (expanded) contract_id Change-Id: I6cab67c2f549e9165dbf2cfd2f4b84daef503dd8	3 years ago
Kirill Solomko	5a18d5fee8	TT#65101 improve expand=all, /api/domains care roles support ... * expand=all now expands by the all resource keys instead of the current form fields, that is to reflect keys that are created manually in the response such as reseller_id or domain_id * ccare roles now correctly support /api/domains Change-Id: I9951bfd97b76186def4dc799c72da44425faea4a	3 years ago
Kirill Solomko	84e698e7ac	TT#176050 adjust subfield expands ... Change-Id: I9826d254cec6f03e7c7bad7122450f4a22125e18	3 years ago
Kirill Solomko	938f6ba05c	TT#176050 add expand support on array of ids ... * it is possible to expand arrays of ids now * add expand support for pbx_group_ids * add expand support for pxb_groupmember_ids * remove _password and _webpassword internally prepared fields from expands by subscriber_id, pbx_group_ids, pbx_groupmember_ids Change-Id: I7651aae4c58d98943e82d1eda6b24d260ff2480a	3 years ago
Kirill Solomko	f311177236	TT#182101 Fix check_allowed_ngcp_types detection ... * use any instead of grep for checking the allowed_ngcp_types array Change-Id: I7382a2a61764a4520cd07d2db67ffa3a5112b378	3 years ago
Kirill Solomko	09b589738d	TT#182101 Phonebook feature is hidden for CE ... * templates now rely on c.config.general.ngcp_type and hide the Phonebook feature everywhere if ngcp_type is CE * introduce "allowed_ngcp_types" config for Controller::API::* that is an array, and when specified, only the ngcp_type roles are allowed, otherwise if not specified all ngcp types allowed (default) * Controller::API::Root: - filter controllers from the documentation rendedring that have allowed_ngcp_type config specified and do not match the current ngcp type * Role::API Role::Entities* - new method check_allowed_ngcp_types() - check_allowed_ngcp_types() is now called in Entities and EntitiesItem auto() and denies to 404 Path not found if the ngcp type does not match * "Phonebook" UI is now hidden for CE * /api/phonebookentries is now hidden for CE Change-Id: I41d4b2f87121f281472be3775b862333923fe37f	3 years ago
Kirill Solomko	68af361060	TT#173700 /api/topuplogs expand related fixups ... * Role::API::Contracts add item_by_id() that calls contract_by_id() * Utils::ProfilePackages::catchup_contract_balances return if $contract object is undef * Form::Topup::Log::contract_id expands into Role::API::Customers instead of Role::API::Contracts Change-Id: Id4fc67b8ea1e91f350d0172aafc2b722f34e61f3	3 years ago
Kirill Solomko	4e309b9ef1	TT#153050 add API soft_expand query parameter ... * soft_expand=1 in API requests when coped with the expand query parameter tells the API to ignore possible expand conflicts (such as field mismatch or a permission issue), returning 200 OK instead of 409 Conflict Change-Id: Ib798aabddb1b4d66fc9708acbf713037696ad600	3 years ago
Kirill Solomko	6a0b5d7a75	TT#149459 expand consider null fields as valid ... * when field expand is requested and the field has "null" value (profile_id,profile_set_id), the field is considered as correct instead of returning 409 Conflict Change-Id: I2dfa918a0b99bff75429baa3e94a734fcb8c17e2	4 years ago
Kirill Solomko	78b80e7064	TT#149459 enhance expand functionality with allowed_roles ... * allowed_roles in the expand defintion restrict the field only to the roles in the allowed list, otherwise the field is not expanded. Change-Id: Ib6f776388457327f2fa85e71deb9591022cee2da	4 years ago
Kirill Solomko	8c556bbe36	TT#149459 improve and automate expand functionality ... * add dictionary support for fields that are expanded if encountered in all endpoints, with a possibility to override it, if defined on the endpoint's field level * move expand definitions from form fields into the Expand dictionary * simplify the expand usage, it now operates only with the <x>_id fields that are returned and visible in the response (e.g. if reseller_id is returned, then ?expand=reseller_id), the returned expand object name is <expand_field_name>_expand the, so in case of ?expand=reseller_id, the returned object will be reseller_id_expand * adapt Role/SystemContacts to work correctly with the expand functionality * expanded fields are returned as <expanded_field_name>_expand Change-Id: I4cab44ede9b40c70a95bbcedc81f58dd1f4e3b67	4 years ago
Kirill Solomko	748ac564b0	TT#148250 enhance API expand functionality ... * add recursive expand support, so it's possbile now to do nested expands as "expand=contract.reseller". * comma separated fields support is preserved and with the enhanced expand "expand=contract.reseller,customer.billing_profile" it enables expanding "contract" with nested "reseller", as well as "customer" with nested "billing_profile". Change-Id: Ie82f5118dc13e57a0397566295644452f29bccde	4 years ago
Flaviu Mates	9bb622a2bc	TT#117153 Improve /api/cfmappings performance ... * switch to 'populate' instead of using 'insert' for each destinations, sources, times, bnumbers, mappings and CF preferences * add API->check_patch_op_add_only - to check if the patch contains only "add" operations * improve /api/cfmappings, if all PATCH operations are "add" then the existing records are not fetched and not recreated, enabling very new mapping inserts Change-Id: I0b4e71565c11771026dbbc000aa57b2a613409fa	4 years ago
Rene Krenn	b6e696621b	TT#132650 /api/provisioning_templates CRUD rail ... the /api/provisionintemplates rail provides the operations to create, edit and delete "provisioning templates" know from the "batch provisioning" feature of admin panel. these templates can also be defined in config.yml, while it is however only possible to edit templates stored in the database. executing a template and/or uploading a .csv for bulk execution will be available in a separate part. Change-Id: If8627327270edfce5bca1be3b1f777c1bd44e90f	4 years ago
Kirill Solomko	cc10506b2e	TT#129162 fix subscriber webpassword field validation ... * 'webpassword' field is now also validated for invalid (non-ascii) characters * Fix multiple APP input field validation erros to comma joined. * Adjust 'webpassword' field validation errors to have better readability when there are multiple validation errors Change-Id: I21536f97a4da78cc5192a3abd8cd5adef1b819ec	4 years ago
Kirill Solomko	2b144caf2c	TT#128605 implement dynamic fields expand ... * add API functionality to request additional data and expand fields in GET methods * syntax: - /api/resource/?expand=all - expands all expandable fields e.g.: customer_id field is expanded and customer internally is queried and returned under "customer" => {...} (the returned data is identical to what /api/customers/id would return) - /api/resource/?expand=reseller_id,customer_id - expands only reseller_id and customer_id fields, if they are expandable - /api/resource/?expand=reseller_id,invalidfield_id - returns the data and expands only fields that are expandable (reseller_id in this case) but if it finds either unknown fields or non-expandable fields, changes HTTP status code to "409 Conflict" * adapt all API endpoints to support dynamic expand fields expanding functionality, however the actual expand for them requires modifying the form fields in the following format: has_field 'contact_id' => ( element_attr => { expand => { class => 'NGCP::Panel::Role::API::CustomerContacts', id_field => 'contact_id', alias => 'contact', fetch => 0, }, }, ); - class - represents the class that should be used by the logic to fetch the relevant data - id_field - which field from the resource needs to be expanded, it should be the "id" field (subscriber_id, domain_id, etc.) - alias - (optional), under which key the fetched data is stored. the field name is used as the key if the option is omitted. - fetch - (optional), if the returned data is under $data->{contract_id} then it will be fetched from there and stored under the key (field name or alias), otherwise the whole retreived data is stored under the key (field name or alias) * adapt /api/autoattendants to use the new approach (old one was expand=1) * currently supported endpoints with expand: - admins - autoattendants - domains - customers - customercontacts - resellers - subscribers Change-Id: Iac53409dad944ed4794039a48dc3a9f6dce25bc1	4 years ago
Kirill Solomko	96f0b07469	TT#116251 validate params page and rows equal 0 ... - a validate logic is added for 'page' and 'rows' query parameters and if either of them equals 0 then 400 Bad Request error is returned Change-Id: Ida850c5d942e51a6328f5df01b8e28d55c32cc56	4 years ago
Rene Krenn	6d89e74253	TT#59803 obfuscate .*password.* fields in request+response body (json) ... Change-Id: I9f457b8cd9b0ff2e94c5795fb9987f97af5ac1f9	6 years ago
Kirill Solomko	9094ec496e	TT#60502 refactor PATCH by value ARRAY ... * PATCH by value when it is an ARRAY now iterates backwards through the current values and it iterates as: for current_values for values_to_remove instead of: for values_to_remove for current_values to preserve indexes order * check and avoid situations when indexes for removal are added multiple times, by marking already added indeses for removal Change-Id: Id50c5ea08ad0f7c626a6b23b288e6d008a19696a	6 years ago
Kirill Solomko	ffdb2d0010	TT#60502 fix PATCH removal by value when multiple elements are removed ... * elements must be removed reversed so last index is removed first, otherwise the list "for removal" becomes out of sync with the "current elements" causing wrong values to be removed, or an out of bounds index array error Change-Id: I9ab9bce8205169bc7841c51f37743ab17946cc11	6 years ago
Rene Krenn	b0465b9f07	TT#59803 obfuscate logged request+response body to protect passwords ... Change-Id: I18003de5a24e39b64cf340ec0d4b758a1a223af9	6 years ago
Rene Krenn	dda5ffe48f	TT#56340 fix subscribers.t not matching obfuscation chars ... Change-Id: I7023d1e23fc397e3556705fc52db50eace6c02b4	6 years ago
Rene Krenn	cae5270af5	TT#56340 panel logline obfuscation ... adds gdpr obfuscation quoting for: + subscriber numbers + subscriber ip addresses + subscriber usernames + any logmessage "DATA": query parameters, form data, response data + subscriber uuid's + call id's + callforward sip uri's the quoting is centralized by $c->qs() ("quote sensitive"), using catalyst plugin mechanism. escape symbols are set to « (\x{ab}) and » (\x{bb}). generate_logfile_data_inventory.pl was modified to mark loglines with "gdpr affected" status, if $c->qs() was used in a log message. Change-Id: I0f42d7992594232ae33e5666b0a64009211c5b76	6 years ago
Irina Peshinskaya	345f89d407	TT#47534 Add configured defailt mime-type to Entities FW ... And enable non_json_data processing for Items PUT Change-Id: Ib87bed6aadce2f9217baa60aedc233364560876a	6 years ago
Irina Peshinskaya	858905035b	TT#50975 Replace "value" logic to "filter" in "remove-by-value" ... PATCH "remove by value" will from now remove values even if provided hash has less keys, than original value Change-Id: I2ee04fa279580d662e167c3a0b160fe29d8d79ef	6 years ago
Irina Peshinskaya	a376cc6619	TT#49027 Fix capabilities api ... Change-Id: I85d3d4e71b745115e18e1530808a3ac84c0917d5	6 years ago
Irina Peshinskaya	39bc7eb75d	TT#48198 Fix order_by functionality in api ... - All item_rs modifications should be done before we get rows, so apply order_by before pager - We can't distinguish if subscriberregistrations really has column or not if we return true for all columns. We will use has_column only in cases when it returns something really meaningful. - Subscriberregistrations can't order by nat and subscriber_id Change-Id: I04b7bb719ee058590a7705c6411cb08bcfb15387	6 years ago
Irina Peshinskaya	39918f4f90	TT#48198 Avoid extra page in no_count mode ... Change-Id: Ia0415cf708bc8d531aaf078dc74d6f81b23b60f2	6 years ago
Irina Peshinskaya	5219cdb244	TT#48198 Fix no_count behavior in API.pm ... Change-Id: Ieb665a7fc82f8442e0b2da351cf92573d433c3b6	6 years ago
Irina Peshinskaya	83d715230e	TT#47963 Fix emailtemplates put and patch ... Change-Id: I26a24c17b42e3f31390de28fd1e5098c6c167861	7 years ago