* log_response: 0 is reworked, it now always logs the response but
in case if it's off the body is not logged.
* adjust error logging, the error in the body is correctly
logged in the response even if log_response: 0
Change-Id: I6dfddeccffb5464b830133235bf63c5e82ae6073
* unhandled internal errors (such as DBI error) are now properly
returned as 500 Internal Server Error instead of 200 OK
Change-Id: Iabc80cd200c9091dd249bc90aa586d5d7b616b69
* the new endpoint accepts new_password fields and enables
for authenticated user a mechanism to quickly change their password
* the global password validation rules are enabled
* returns 204 No Content
* if user's password is expired, the endpoint is the only accessible
for the user to change the password and unlock other endpoints.
* a few fixes in validate_password() to correctly fetch provisioning
subscriber for password change scenarios and webpassword field
Change-Id: I906fcfe5c780b850d322b46b445b54c054767673
* device_id, profile_id (device related), config_id are now
expandable in the respective /api/pbxdevice* endpoints.
* fix expand collection logic to avoid ambiguous 'order by id'.
Change-Id: I0b1f4b3da093fb04a30e5097881af6131c1afe46
* UI: password are now validated against
$c->config->{security}{password}{web_max_age_days} (unless it's 0)
and if the password is expired the user is redirected automatically
to /changepassword page, and after successful password change back
to the original page.
* API: if password is expired all API requests will be returning
403 Forbidden "Password expired", except PUT/PATCH to /api/admins
or /api/subscribers with the new password in place.
* successful login on the UI now redirects to /dashboard instead of
/ (to prevent unintended redirect to v2)
Change-Id: I075f8e17cc9b0658d6b3b3d526ca5b379d050ce4
* UI and API parts are now under license control
* new Util::License::get_license($c, $name)
- fetches license status by name (1 if enabled, and also if
/proc/ngcp/check if 'ok')
* add Catalyst::Plugins::NGCP::License with license($name)
to fetch valid license by name from anywhere using
$c->license('pbx') or from the templates using c.license('pbx').
It internally uses Util::License::get_license($c, $name)
* License::get_license_status($c) now requires $c as first argument
as well logs license status check errors.
* new ActionRoles::License that enables usage of
:Does(License) RequiresLicense('pbx') LicenseDetachTo('/denied_page')
in the Controller chains
* Add license control for UI elements and return 403 Forbidden
if a resource is covered by licenses and the license is not active
* Hide UI elements if a license is not active
* API/Entities/Entities new $c->set_config key:
- per endpoint:
$c->set_config({
required_licenses => [qw/pbx device_provisioning/]
}
- or per method:
$c->set_config({
required_licenses => {
POST => [qw/pbx device_provisioning/]
}
}
}
* In case if an API endpoint does not have a license:
403 Forbidden "Invalid license" reply is returned.
* Add license based restrictions to API endpoints
* /api documentation:
- completely hide endpoints that do not have an active license
- hide only methods that does not have an active license
Change-Id: Iba45fc5068b02306a617fed7b5405f2210574b61
* add search by not_null for fields that ends with _id
and integers, that adds "IS NOT NULL" to the SQL search query
by the field.
* null and not null values are expected now as $null and $not_null
to avoid conflicts when user searches by null strings that may
be a valid case.
Change-Id: I8e8b8c9060e985dfe2b94cbfcca1587f05477fe9
* search by value null is internally translated to undef
to say search for records without a value, same way as
it already works for NULL value, just null is more expected
and easier to use by clients as it's how its represented in the
resource
Change-Id: Ia8a75bad95a34dd8167162d0f09a1ec7c4056105
* in general expand collection now performs only 1 sql
query to per expand field to fetch all items by the ids
instead of fetching them for every single collection item,
that should significantly increase performance in case
of large databases and reduce work for the database
* introduce $c->stash->{expand_cache} that contains
cached data for the expand fields to avoid multiple
same calculations
* expand_field() and expand_field_data() have been reworked
to support expand_cache
* new method expand_prepare_collection() is called for all
API GET collection methods before preparing resource hal
fields, to change the expand logic to only cache the data
instead of fetching it from the database
* new method expand_collection_fields() that is called in all
API GET collection methods after the @embedded data is prepared
to finalise the expand collection fields
* for expand collection there is only 1 SQL request per expand field
that fetches all items -in [ids to expand] and then the
expand_collection_fields() uses the cached items_by_ids with
O(1) fetch from the cache by id
Change-Id: Ie7c6115472878febf0d8c9b4d833f5c23b15c78b
s/paramaters/parameters/
s/wihout/without/
s/If it mathes/If it matches/
s/ in in / in /
s/the the/the/ (note: fixes translation issue which was introduced with
commit 2b7a1a33 and fixed in 072e897c for the original, but missed
for the german translation)
Change-Id: I4186bf3f42b1fac11bb7d7fe801f860d2d59adc1
* clear errors in Controller::API::Root::end so that they are
not interfere with generated ones
* API::error(): add $c->stash->{is_api_error_response} = 1
to track it in log_response()
* API::log_response(): distinguish and correctly correct errors
for cases when is_api_response == 1 and also when there were
$c->error('error') calls in the code, to log them correctly
in both cases
Change-Id: Id922b8219832c1b99815d9d608309035b9b25cff
* $c->error array now contains the $message as the first element
instead, so that it's possible to obtain all the error data in
the code when fetching it from @{$c->error}. the first element
is not logged in the error log.
* api_response $c->response_body part is now stored in MSG= and
possible errors / other log data is now stored in LOG= to:
- reduce amount of log lines when an API error response occurs
from 2 to 1
- the message part usually contains either HTTP response
message (e.g. Internal Server Error) or a validation message
string, so it belongs to the MSG= part of the log line, where
as the internal log data is more related to the LOG= part
- both MSG= and LOG= parts are escaped for GDPR related obfuscation
* Utils::Messag::info(): $msg is now also obfuscated if it's detected
as a reference (also because logging is moved for the API part to
$msg), as well as truncated for possible new-line char and
white-spaces.
Change-Id: I3b670b2251ec3060037ed6863f18d95975120b8d
* Role/Entities*: POST/PUT/PATCH/DELETE methods changes:
- support deadlock detection and transaction retry
(2 retry attempts at the moment)
- improve transaction control, use local $guard
instead of saving the ref to $c->stash, as in that
case it went out of scope too late and also reported
an error message into the log about abnormal $guard out
of scope interruption
- move all non transaction related code outside
of the scope
- add error handling when methods such as update_item,
and a like do not return the expected data, instead of
simply going out of scope and resulting in an uncontrolled
reply
* Role/API:
- rework transaction control:
+ get_transaction_control() is renamed to start_transaction()
to better reflect what it does
+ complete_transaction() is renamed to commit_transaction()
+ remove unused %params arg
+ pass $guard into commit_transaction() instead of having it
stored as $c->stash->{transaction_guard) that caused the $guard
ref to be destroyed much late than expected
(there was also a typo as transaction_quard,
which is not relevant anymore with the changes
+ add check_deadlock() that is invoked when an exception is caught
or an $c->errors contain an error, and if the error message
represents a transaction error, the transaction block is
re-invoked via "goto TX_START"
- rework error():
+ it now accepts args as following:
($self, $c, $code, $message, @errors)
# code -> returned as HTTP code in the reply
# message -> returned as HTTP message in the reply
# errors -> contain errors for internal logging, last element often contains a DBIx exception
+ populates all @errors into $c->error so they are available on
demend in the code via $c->error or $c->last_error
+ $c->log->error is not invoked now as the errors become printed
in log_response()
- log_response() now prints collected errors from $c->error correctly
as a separate log line, that is alike to the other api logs so that
those can be looked up by the request's tx_id, also all errors are
now printed only into api.log
* Adjust all $self->error() calls in catch($e) to include $e as the last
argument, as well as the duplicate $c->log->error is removed from
those ocassions
* Remove all $c->log->error() calls as they are replaced with either
$self->error() (that logs it correctly into api.log) or
$c->error('err') that also adds it correctly into api.log
* API::CallForwards: rework to use Entities/EntitiesItem
* API::Contracts: rework POST to use Entities
* API::PeeringGroups: rework POST to use Entities
* API::SubscriberRegistrations: rework POST to use Entities
* API::RewriteRuleSets: improve create_item() functionality
* Utils/Message: add 'api_retry' log type
* $c->session->{api_request_tx_id} is changed to
$c->stash->{api_request_tx_id} because sometimes the session
ref is different and a different tx_id becomes used
Change-Id: I633ce7a8047b1bf00a2f6889003088edf0825dcd
all api query param filters that were harmonized to "wildcard"
also appended leading+trailing wildcard (this was from the original
idea to make adminUI and legacy panel search behave the same).
Change-Id: I017473b84f2844d14a11b4ec8b437df5335c24d0
the /api/callists rail supports the "call_id" query parameter to match calls with a callid prefix. this filter also adds an implicity ordering ORDER BY length(call_id) ASC, 'start_time' ASC.
the /api/callists api rail also requires the query parameter "susbcriber_id", which renders a (fast) compound SQL query to list matching incoming (query1) and outgoing calls (query2) using UNION ALL (instead of a slow OR query).
query1 UNION ALL query2
this is supported by the https://metacpan.org/pod/DBIx::Class::Helper::ResultSet::SetOperations module, which however generates invalid SQL syntax if query1 or query2 contains an ORDER BY.
this is exactly what caused the 500 error of the customer who applied both "call_id" and "susbcriber_id" parameter in the request at the same time.
... ORDER BY length(call_id) ASC, `start_time` ASC UNION ALL SELECT `me`.`id`...
the error happens randomly, because query parameters are stored in a hash (which by definition has no deterministic order of the entries). when the "call_id" parameter is applied at last, it worked as expected.
so the issue can be adressed by strictly ordering the UNION ALL result, and prohibit ORDER BY in query1 and query1. the latter was added already with commit b2dfe28eed, and could be hotfixed.
however, the ORDER BY of "call_id" query param is lost, and should be properly handled by forcing strict ordering of how query parameters are applied.
the fix will introduce paramater ordering according to their order of definition in the code.
Change-Id: I165d341b5c20e9bb750bd1fba88c836b393e80bd
- all standard LIKE search are migrated
- will avoid LIKE unless a pattern (* wildcard) is
used as a search term. this encourage db index
usage, will be faster
- supports wildcard escape sequence \\*
- harmonize swagger UI descriptions of filters
Change-Id: Iea155871c9be6c284e6970a562d4e6af73fedc4b
filtering an Entity-Attribute-Value model cannot be done as
simple conjunctions, but requires either INTERSECT set
operation, or joining the same table multiple times.
Change-Id: I5ce1ae1ece9406b6610487654f09d768a233b122
various api rails will need to support ?caller= and ?callee=
url query parameters.
since this involves SQL queries against potentially large
database tables, special care is taken with wildcard search
to prevent slow queries:
- the ?wildcards=true query parameter has to be specified
to accept search patterns that contain wildcard symbols,
so wildcards are not accepted by default.
WARNING: a search string with a leading wildcard will always
force a *slow* full db table scan!
- the * symbol is used as a wildcard symbol
- \ (backslash) is used as escape character to search for
a literal '*'
Change-Id: Ie6065b0cfa883f7963e1dc8259fffea9a1edfdfe
* uses prefix 'me.' for query types search_eq and search_like
in the where condition to avoid ambiguity if the search_rs
has joins that with the same column names
Change-Id: I90fef80970aa4415480b00bbed2fb9fbee1f1ccc
* the same key is now expanded only once to use the same
data ref otherwise expand like
contract_id.contact_id,contract_id.billing_profile_id
makes it so that 2nd contract_id would overwrite the already
existing (expanded) contract_id
Change-Id: I6cab67c2f549e9165dbf2cfd2f4b84daef503dd8
* expand=all now expands by the all resource keys instead
of the current form fields, that is to reflect keys that
are created manually in the response such as reseller_id
or domain_id
* ccare roles now correctly support /api/domains
Change-Id: I9951bfd97b76186def4dc799c72da44425faea4a
* it is possible to expand arrays of ids now
* add expand support for pbx_group_ids
* add expand support for pxb_groupmember_ids
* remove _password and _webpassword internally prepared fields
from expands by subscriber_id, pbx_group_ids, pbx_groupmember_ids
Change-Id: I7651aae4c58d98943e82d1eda6b24d260ff2480a
* templates now rely on c.config.general.ngcp_type and hide
the Phonebook feature everywhere if ngcp_type is CE
* introduce "allowed_ngcp_types" config for Controller::API::*
that is an array, and when specified, only the ngcp_type
roles are allowed, otherwise if not specified all ngcp types
allowed (default)
* Controller::API::Root:
- filter controllers from the documentation rendedring
that have allowed_ngcp_type config
specified and do not match the current ngcp type
* Role::API Role::Entities*
- new method check_allowed_ngcp_types()
- check_allowed_ngcp_types() is now called in Entities and
EntitiesItem auto() and denies to 404 Path not found if
the ngcp type does not match
* "Phonebook" UI is now hidden for CE
* /api/phonebookentries is now hidden for CE
Change-Id: I41d4b2f87121f281472be3775b862333923fe37f
* Role::API::Contracts add item_by_id() that calls contract_by_id()
* Utils::ProfilePackages::catchup_contract_balances return if
$contract object is undef
* Form::Topup::Log::contract_id expands into Role::API::Customers
instead of Role::API::Contracts
Change-Id: Id4fc67b8ea1e91f350d0172aafc2b722f34e61f3
* soft_expand=1 in API requests when coped with the expand
query parameter tells the API to ignore possible expand
conflicts (such as field mismatch or a permission issue),
returning 200 OK instead of 409 Conflict
Change-Id: Ib798aabddb1b4d66fc9708acbf713037696ad600
* when field expand is requested and the field has "null" value
(profile_id,profile_set_id), the field is considered as correct
instead of returning 409 Conflict
Change-Id: I2dfa918a0b99bff75429baa3e94a734fcb8c17e2
* allowed_roles in the expand defintion restrict the field
only to the roles in the allowed list, otherwise the field
is not expanded.
Change-Id: Ib6f776388457327f2fa85e71deb9591022cee2da
* add dictionary support for fields that are expanded
if encountered in all endpoints, with a possibility
to override it, if defined on the endpoint's field
level
* move expand definitions from form fields into the
Expand dictionary
* simplify the expand usage, it now operates only with the
<x>_id fields that are returned and visible in the response
(e.g. if reseller_id is returned, then ?expand=reseller_id),
the returned expand object name is <expand_field_name>_expand
the, so in case of ?expand=reseller_id, the returned object
will be reseller_id_expand
* adapt Role/SystemContacts to work correctly with the expand
functionality
* expanded fields are returned as <expanded_field_name>_expand
Change-Id: I4cab44ede9b40c70a95bbcedc81f58dd1f4e3b67
* add recursive expand support, so it's possbile now to do
nested expands as "expand=contract.reseller".
* comma separated fields support is preserved and with the enhanced
expand "expand=contract.reseller,customer.billing_profile" it enables
expanding "contract" with nested "reseller", as well as
"customer" with nested "billing_profile".
Change-Id: Ie82f5118dc13e57a0397566295644452f29bccde
* switch to 'populate' instead of using
'insert' for each destinations, sources,
times, bnumbers, mappings and CF preferences
* add API->check_patch_op_add_only - to check if
the patch contains only "add" operations
* improve /api/cfmappings, if all PATCH operations
are "add" then the existing records are not
fetched and not recreated, enabling very new mapping
inserts
Change-Id: I0b4e71565c11771026dbbc000aa57b2a613409fa
the /api/provisionintemplates rail provides the
operations to create, edit and delete "provisioning
templates" know from the "batch provisioning" feature
of admin panel.
these templates can also be defined in config.yml,
while it is however only possible to edit templates
stored in the database.
executing a template and/or uploading a .csv for bulk
execution will be available in a separate part.
Change-Id: If8627327270edfce5bca1be3b1f777c1bd44e90f
* 'webpassword' field is now also validated for invalid
(non-ascii) characters
* Fix multiple APP input field validation erros to comma joined.
* Adjust 'webpassword' field validation errors to have better
readability when there are multiple validation errors
Change-Id: I21536f97a4da78cc5192a3abd8cd5adef1b819ec
* add API functionality to request additional data
and expand fields in GET methods
* syntax:
- /api/resource/?expand=all - expands all expandable fields
e.g.: customer_id field is expanded and customer internally
is queried and returned under "customer" => {...}
(the returned data is identical to what /api/customers/id
would return)
- /api/resource/?expand=reseller_id,customer_id - expands
only reseller_id and customer_id fields, if they are expandable
- /api/resource/?expand=reseller_id,invalidfield_id -
returns the data and expands only fields that are expandable
(reseller_id in this case) but if it finds either unknown
fields or non-expandable fields, changes HTTP status code
to "409 Conflict"
* adapt all API endpoints to support dynamic expand fields expanding
functionality, however the actual expand for them requires modifying
the form fields in the following format:
has_field 'contact_id' => (
element_attr => {
expand => {
class => 'NGCP::Panel::Role::API::CustomerContacts',
id_field => 'contact_id',
alias => 'contact',
fetch => 0,
},
},
);
- class - represents the class that should be used by the logic
to fetch the relevant data
- id_field - which field from the resource needs to be expanded,
it should be the "id" field (subscriber_id, domain_id, etc.)
- alias - (optional), under which key the fetched data is stored.
the field name is used as the key if the option is omitted.
- fetch - (optional), if the returned data is under
$data->{contract_id} then it will be fetched from there and
stored under the key (field name or alias), otherwise the whole
retreived data is stored under the key (field name or alias)
* adapt /api/autoattendants to use the new approach (old one was expand=1)
* currently supported endpoints with expand:
- admins
- autoattendants
- domains
- customers
- customercontacts
- resellers
- subscribers
Change-Id: Iac53409dad944ed4794039a48dc3a9f6dce25bc1
- a validate logic is added for 'page' and 'rows'
query parameters and if either of them equals 0
then 400 Bad Request error is returned
Change-Id: Ida850c5d942e51a6328f5df01b8e28d55c32cc56
* PATCH by value when it is an ARRAY
now iterates backwards through the current values and
it iterates as:
for current_values
for values_to_remove
instead of:
for values_to_remove
for current_values
to preserve indexes order
* check and avoid situations when indexes for removal are
added multiple times, by marking already added indeses for removal
Change-Id: Id50c5ea08ad0f7c626a6b23b288e6d008a19696a
* elements must be removed reversed so last index is removed first,
otherwise the list "for removal" becomes out of sync with the
"current elements" causing wrong values to be removed, or an
out of bounds index array error
Change-Id: I9ab9bce8205169bc7841c51f37743ab17946cc11
adds gdpr obfuscation quoting for:
+ subscriber numbers
+ subscriber ip addresses
+ subscriber usernames
+ any logmessage "DATA": query parameters, form data, response data
+ subscriber uuid's
+ call id's
+ callforward sip uri's
the quoting is centralized by $c->qs() ("quote sensitive"), using
catalyst plugin mechanism.
escape symbols are set to « (\x{ab}) and » (\x{bb}).
generate_logfile_data_inventory.pl was modified to mark loglines
with "gdpr affected" status, if $c->qs() was used in a log message.
Change-Id: I0f42d7992594232ae33e5666b0a64009211c5b76
PATCH "remove by value" will from now remove values even if provided hash has less keys, than original value
Change-Id: I2ee04fa279580d662e167c3a0b160fe29d8d79ef
Kirill Solomko
Michael Prokop
Rene Krenn
Flaviu Mates
Irina Peshinskaya