query refactoring an rowcount clipping for UI datatables
that are slow when using millions of subscribers:
contacts
customers
contracts
subscribers
billing profiles
billing networks
billing profile packages
Change-Id: Ia50e3aa52684772548569b6908f0cbc08395a5a7
* add recursive expand support, so it's possbile now to do
nested expands as "expand=contract.reseller".
* comma separated fields support is preserved and with the enhanced
expand "expand=contract.reseller,customer.billing_profile" it enables
expanding "contract" with nested "reseller", as well as
"customer" with nested "billing_profile".
Change-Id: Ie82f5118dc13e57a0397566295644452f29bccde
* "subscriber" role can now retrieve own autoattendant data. This is needed for the CSC interface.
Change-Id: Id10b302205fe458d5793ae8f7bd9201233f9a0d4
* Admins with is_system and is_superuser are able to see the items for all roles.
* Admin is able to see own journal.
Change-Id: I3e5d459b08ff7ef218220f1ae11974351121c489
the POST /api/provicioningtemplates/<reseller>/<template>
request will accept text/csv content type to provision
many susbcribers at once.
Change-Id: I59079ba8f2bacc0ce2b1367d2bd1a7251cf4763c
provisioning templates with their dynamic forms
can be executed by a entityitem POST request, ie.
POST /api/provicioningtemplates/<reseller>/<template>
or
POST /api/provisioningtemplates/<readonly template>
Change-Id: I77f6c9d42e1afdb49635d3f11e4d73bcf6269605
* vmnotify() now accepts cli and uuid arguments
* API handling of voicemails is now improved to:
- send a notify if the item's INBOX/Old has been changed
- correctly process DELETE to send vmnotify after the
item's removal
Change-Id: Ic00ae825cf091bce273e55aa37cd0a7ac80d8b0f
* improve select from voicemail_spool to avoid
sequental scan with like '%..'
* select now fetches all messages count
* add old messages count support
* old/new messages are reported as 0 if not returned
from voicemail_spool
Change-Id: I11ac1a407e8d22fe828a17cda55aa3298c6e6f02
* non-existing hal resource key was used in the debugging string
causing the item operations, such as PATCH to produce 500 error
Change-Id: I9e53dcf8b4b21385b9b1b877d87319023222f620
* domains do not use billing.domain_resellers table anymore
but instead the new domains.reseller_id field. That is
to remove the unneeded many<>many relation through the
additional table where the actual logic only supports
one(reseller) to many(domains) relation
Change-Id: I1b681543baf1901f19e10c2f6210e4cf6eeb8fbe
* move to_log() and data_to_str() methods to the parent
VendorREST module
* rename send_request() to send_http_request()
* use common send_http_request method for HTTP requests
dispatching
* improve logging
* improve error handling
Change-Id: I403aa8053e4abfaf6992b62809f15ab72e3a06fc
the form used for the /api/ncoslnpcarriers rail
uses the same technique to expose a JSON field
"carrier_id" for the table column "lnp_provider_id",
also found in the /api/lnpnumbers rail implementation.
it however did not work until now, because the
fieldname in the render_list did not match.
Change-Id: I45dca22bab73e16b538de7ea1d540aa1383fb56d
Otherwise PBX devices config geenration failed with error:
> 500 - error creating template:
> error processing template, type=undef,
> info='Can't use string ("") as a HASH ref while "strict refs"
> in use at /usr/share/perl5/NGCP/Panel/Template/Plugin/DeviceConfig.pm line 50. '
It happens because no preferences access defined for 'bootstrap' PBX
access (port 1445), they are available for 'config' access only (port 1444).
Change-Id: Id2c1478678265144b8ddb83504a0d3c9cec16563
* switch to 'populate' instead of using
'insert' for each destinations, sources,
times, bnumbers, mappings and CF preferences
* add API->check_patch_op_add_only - to check if
the patch contains only "add" operations
* improve /api/cfmappings, if all PATCH operations
are "add" then the existing records are not
fetched and not recreated, enabling very new mapping
inserts
Change-Id: I0b4e71565c11771026dbbc000aa57b2a613409fa
The latest version of the 'ul.add' RPC command of Kamailio
has 11 parameters instead of 9, we have to fix that.
Change-Id: Ic70bde77eada095ddf127e18f3ef7e5b3d91478f
the /api/provisionintemplates rail provides the
operations to create, edit and delete "provisioning
templates" know from the "batch provisioning" feature
of admin panel.
these templates can also be defined in config.yml,
while it is however only possible to edit templates
stored in the database.
executing a template and/or uploading a .csv for bulk
execution will be available in a separate part.
Change-Id: If8627327270edfce5bca1be3b1f777c1bd44e90f
the G2k/LIMA implementation seems to have our error
messages hardcoded in their logic. it stareted to
fail since we added logline obfuscation, which was
also added to response messages in this place.
Change-Id: I36c2a74ed17db7013e692d7cbdccf0dbd44e814e
The ngcp-panel v1 codebase uses 'back=' GET parameters to record
all the navigation path and store it into the session array:
'$c->session->{redirect_targets}'.
On switch from v1 to v2 using the link 'GO TO NEW ADMIN PANEL',
the function 'login_to_v2' is not using the concept of 'back=' GET param,
but ngcp-panel still receives and stores the last value with 'empty' path:
> $VAR1 = bless( do{\(my $o = 'https://x.x.x.x:1443/')}, 'URI::https' );
> $VAR2 = bless( do{\(my $o = 'https://x.x.x.x:1443/subscriber/155/details')}, 'URI::https' );
> $VAR3 = bless( do{\(my $o = 'https://x.x.x.x:1443/subscriber')}, 'URI::https' );
> $VAR4 = bless( do{\(my $o = 'https://x.x.x.x:1443/dashboard')}, 'URI::https' );
The navigation above is a recorded browsing path on v1 (in a reverse order):
- login to ngcp-panel (dashboard page is opened),
- open 'Subscribers'
- open details for some subscriber with id 155
- open subscriber preferences
- click on link 'GO TO NEW ADMIN PANEL'.
As a result user is still located on the same page "Preferences",
but not on v1 but v2 interface. The empty value is inserted into
the array '$c->session->{redirect_targets}' (which is wrong).
The empty path 'https://x.x.x.x:1443/' brakes v2 navigation
for v1 'Back' button inside iframe.
It causes loading of iframe inside iframe,
which happens on v2 due to list of redirections:
- clicking on v1 button 'back' inside v2 iframe requests https://x.x.x.x:1443/back
- it triggers navigation to the top element array 'https://x.x.x.x:1443/' which is wrong/corrupted.
- loading '/' cause 302 redirect to '/v2/' (as 'v2' is a default UI for mr10.0+)
- loading '/v2/' inside iframe cause the issue with 'v2' content inside 'v2' iframe.
This is a commit to prevent inserting an empty 'back_uri' into
the session array '$c->session->{redirect_targets}'.
Change-Id: I69df4320fa8cde4d23a7d9dd18ffb5eb06ee8df1
The modern browsers (except Safari) supports 'Sec-Fetch-Dest':
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Sec-Fetch-Dest
In theory we can remove the current 'framed session' storage completely,
but Safari will not be supported. Let's expand the current logic to provide
extra protection here to backport the commit to mr9.5+.
Change-Id: I9c070f77f427c81581f4d9ceeb1a57b274d77819
* kamailio.sems_registrations now uses
kamailio.lcr_gw.id -> peer_host_id
kamailio.subscriber.id -> subscriber_id
for foreign keys consistency withing the same database
Change-Id: I4722729fdee07eb2a153473e85d64faa46eedca5
* webpassword is not correctly removed based
on length, and remain visible when in plain-text
or empty (unset)
* config->security->password_(sip|web)_expose_subadmin
now only affects subscribers under the same customer
that are not this subscriber admin
Change-Id: I329e0f1ad97dd513a33e3652ed03b4a43a95ed04
We need to de-register all subsequent peering hosts located under
the peering group being deleted.
Change-Id: I50bc25932e59d6b918f65c6525f2631cb9868fe4
We need to add an improvement, which fulfills the work of the
registration mechanism for SIP peerings.
Not only do we add here a registration tirggering for peerings,
but also a 'type' is introduced, which is mostly needed for XMLRPC
commands being sent towards SEMS, to let it understand to whom the $sid is related to.
Plus a list of improvements:
- de-register peering host on deletion ;
- de-register peering host on disable / register of peering host on enable ;
Change-Id: I035dfadf6709acb4d106a70f6124f024e719044f
Throw descriptive error for attempt to create Customer with Template
that does not belong to Contact’s Reseller. Cover the case when the
Template belongs to System Contact (with no Reseller).
Error example:
'subscriber_email_template_id' with value '1' does not belong to
Reseller '1' that is assigned to Customer's Contact '1'
Change-Id: Iffcef0339afc4490ecba81d4667cbb9225766af4
fee matching works in 2 attempts:
1. try to find a fee matching <user@domain>
2. if none found, try to find a fee matching <user>
the billing_profile record will get a new field
"ignore_domain". if set to "1", step#1 above
should be skipped.
Change-Id: I7f0a8f1dd8d5699ad4e2b8242c68fb9f75fc1bff
* password field is virtual and needs to be transformed
into saltedpass, restored the removed logic that does that
in process_form_resource()
Change-Id: I8baabbef2bdb46db850e12b6e0c638ca5c3deddf
A feature request to have an opportunity to add a default action slot,
in case a caller hangs on the AA and doesn't pick anything.
Main change is related to sql structure of 'provisioning.voip_pbx_autoattendants' table,
which now has 'voip_pbx_autoattendants.choice' as VARCHAR(16) instead of TINYINT.
Also adding a label 'default' for the drop-down menu.
Change-Id: I6382a68c8258094a2bf2bf9c71407092860f5e4d
The value of "interval_free_time" preference is specified in seconds.
To avoid confusion, use the term "free calling time" rather than "free
minutes" in tooltips.
Change-Id: Ia4bd507e4a2281723fd2168a34b2178bf404f867
* 'webpassword' field is now also validated for invalid
(non-ascii) characters
* Fix multiple APP input field validation erros to comma joined.
* Adjust 'webpassword' field validation errors to have better
readability when there are multiple validation errors
Change-Id: I21536f97a4da78cc5192a3abd8cd5adef1b819ec
* webpassword field was unconditionally deleted
in API GET and DELETE methods, it now relies
on resource_from_item for the common approach
Change-Id: I703158fd2022b49a49470db28cb22f37e613f841
* 'administrative' field is read only for susbcriberadmin
role and that caused it to be removed from the final
update 'resource', setting it to 0 if not existed.
now the 'administrative' field is only changed in the database
if it's defined in the 'resource'.
Change-Id: I50738a77052c2163b19b2a42293c7a00e2780bc3
* PATCH: password fields are not removed when
resource is created for apply_patch(), they
are removed under the same condititions later
when hal is generated, that is to ensure that
admin users without the 'show_passwords' flag
as well as subscribers will not run into situation
when they use PATCH and cannot apply it for
"path": "/password" or/and "path": "/webpassword",
as they were removed before apply_patch()
* rework encrypted webpassword detection.
webpasword is detected as encrypted if its length
is 54 or 56 and it contains at least one '$' char,
there is a chance for false positive detection when
a user provides with a plain-text password with the
same pattern but it's very unlikely, as well as
since mr8.5 webpasswords are expected to be encrypted,
and moreover worth case scenario is that the
plain-text password will not be returned to the user
Change-Id: I8ea739cbf728b2134f3ce00cee29da42ab3fb4a3
* add API functionality to request additional data
and expand fields in GET methods
* syntax:
- /api/resource/?expand=all - expands all expandable fields
e.g.: customer_id field is expanded and customer internally
is queried and returned under "customer" => {...}
(the returned data is identical to what /api/customers/id
would return)
- /api/resource/?expand=reseller_id,customer_id - expands
only reseller_id and customer_id fields, if they are expandable
- /api/resource/?expand=reseller_id,invalidfield_id -
returns the data and expands only fields that are expandable
(reseller_id in this case) but if it finds either unknown
fields or non-expandable fields, changes HTTP status code
to "409 Conflict"
* adapt all API endpoints to support dynamic expand fields expanding
functionality, however the actual expand for them requires modifying
the form fields in the following format:
has_field 'contact_id' => (
element_attr => {
expand => {
class => 'NGCP::Panel::Role::API::CustomerContacts',
id_field => 'contact_id',
alias => 'contact',
fetch => 0,
},
},
);
- class - represents the class that should be used by the logic
to fetch the relevant data
- id_field - which field from the resource needs to be expanded,
it should be the "id" field (subscriber_id, domain_id, etc.)
- alias - (optional), under which key the fetched data is stored.
the field name is used as the key if the option is omitted.
- fetch - (optional), if the returned data is under
$data->{contract_id} then it will be fetched from there and
stored under the key (field name or alias), otherwise the whole
retreived data is stored under the key (field name or alias)
* adapt /api/autoattendants to use the new approach (old one was expand=1)
* currently supported endpoints with expand:
- admins
- autoattendants
- domains
- customers
- customercontacts
- resellers
- subscribers
Change-Id: Iac53409dad944ed4794039a48dc3a9f6dce25bc1
* Login CSC v2 button is shown on the subscriber's master
data page if www_admin.http_csc.csc_js_enable == 1 or 2
* When the login is triggered an auth token
is generated internally followed by a redirect to
CSC as /?a=auth_token
* move generate_auth_token() into Utils/Auth
* improve generate_auth_token() arguments support
* add /api/authtokens error handling
Change-Id: Idd65400bf8ce6ce48979c736f6a199fb567ffaa4
It is much more usable to see the debug information as URLs:
> Jul 22 08:24:53 sp1 ngcp-panel: DEBUG: *** New GET request on path: /
> Jul 22 08:24:53 sp1 ngcp-panel: DEBUG: *** New GET request on path: /subscriber
> Jul 22 08:24:53 sp1 ngcp-panel: DEBUG: *** New GET request on path: /subscriber/ajax
Instead of Catalyst oriented way:
> Jul 22 08:24:53 sp1 ngcp-panel: DEBUG: *** New GET request on path:
> Jul 22 08:24:53 sp1 ngcp-panel: DEBUG: *** New GET request on path: subscriber
> Jul 22 08:24:53 sp1 ngcp-panel: DEBUG: *** New GET request on path: subscriber/ajax
Change-Id: I38699152e232c5f5aa2ef218db9bf61c692bbf33
It was close to impossible to read ngcp-panel debug log due to:
* missing clear marker of the start reuqest processing, use '***' once only
* some personal markers (like '+++++++') have been removed as they have no
meaning for other developers. Let's remove the personal markers and work to
make the panel debug log well readable for all developers.
Change-Id: I69faff3ab2258fc156e88c7b8da0edfef14c3e6e
* /login_jwt is now the only endpoint to issue JWT tokens
* JWT token admin/subscriber is provided based on the
NGCP_REALM/NGCP_API_REALM fcgi env values
(e.g.: https://localhost:1443/login_jwt = admin JWT token and
https://localhost/login_jwt = subscriber JWT token)
* Authorization: Bearer a= prefix is deprecated
* Clients cannot use subscriber JWT token to access admin
NGCP_API_REALM https://localhost:1443/api/...
and vice-versa
Change-Id: I46edf4c7aaf7bb835dc4ac6b7535aa2d6b5ac136
Add strings that were collected by fixed ngcp_panel_dump_db_strings.pl
from Forms into i18n source file messages.pot.
Change-Id: I92157059b48bdfab94405b674f9a7300c5a9a91d
Fix ngcp_panel_dump_db_strings.pl to load Forms modules properly.
Make modules not to fail when they are called without ctx context.
Change-Id: I695978b1831068f37198bce9f6a5b9e406d79ede
* the extra packing of the secret key during encode/decode
conflicts with the API v2 implementation
* move JWT "typ" from the payload to the header
Change-Id: Ica5822d810d6eaf7b3ae017f7037f25637b6f861
* unify the logic and use uuid in /api/cfdestinationsets and
/api/cfmappings if the subscriber's primary number is empty
Change-Id: Ibdb7e2a2bcd33165be5ec1d92c4f1df24f59405a
* the endpoint will receive "type" (expires|onetime)
and "expires" (positive integer representing seconds)
* type will define the expiray method for the token;
onetime: the token expires as soon as it's used, or
after "expires" seconds if not used
expires: the token can be used multiple times until
it expires according to the "expires" param value
* login_jwt endpoint for generating the JWT token for
subscribers has been enhanced to accept the "token"
param, containing the token generated using the
/api/authtokens endpoint
* admin_login_jwt endpoint for generating the JWT token
for admins has been enhanced to accept the "token"
param, containing the token generated using the
/api/authtokens endpoint
* login_jwt and amin_login_jwt will respond with 403
"Forbidden" if the token role stored in Redis does
not match the role of the user that generated it
* /api/authtokens is hidden from documentation for now
Change-Id: I4eb76c2b08f2e24774fa84ba0ccf7412ce8670e8
* add additional centralised checks for inactive and read_only users.
* use_userdata_from_session=0 now for all auth realms to cause the data
re-fetched from the database, to avoid scenarios when a user is set
as inactive or read_only and UI keeps using the cached data.
the change only affects cookie and JWT subscriber based sessions
as in all other cases, the auth data is fetched from the storage regardless.
* add is_active=1 flag for the internal 'system' role, as otherwise
access would be permanently denied for it.
* default 403 error for denied api requests is changed to "Forbidden"
instead of "Forbidden path".
Change-Id: I1d6d3c765ca8e017e11845c1f5260243a3963c3b
Otherwise v1 session (cookies-based) is not created and v2 CSC
cannot load v1 "iframe" content. Also it was impossible to switch
between v1 and v2 CSC.
Change-Id: Iab3fac6e8da6e5a2e81a0626a3db6faa3e011396
* add quotation with ~ for square brackets ([])
comming from the database, as it is needed for I18N
Change-Id: Ia1253e90d47858a930a4a9569c2d27993a0cd4bc
* 'umploadtmp' option is now defined in Panel->config
and taken from ngcp_panel.conf otherwise falls back to /tmp
Change-Id: I836f2d7dd346bd1853c513dacd2ab5caffbcdf3e
* /api/callists "type" field now contains the same call type
value as it is in the database, for flexibility so that
customers can implement their own logic around it, as well as
filtering by the query parameter "type" works as expected.
* For history: in /api/callists in case of direction "in", the type
was replaced with type=call regardless of the what was in the database
Change-Id: I1174b34747fe1b739cd6bfc050911c58c4b0964a
* sip_lcr_reload is now called after "commit" in all API endpoints,
to correctly reflect updated DB changes. It was correct in
/api/peeringrules POST but not correct in DELETE, as well as
also not correct in /api/peeringservers and /api/peeringgroups
* sip_domain_reload does not check if the domain is successfully
reload in kamailio proxy as is logic is redundant, it fails
however if domain reload XMLRPC request failed on any available
proxy servers. Another reason is by default tcp_conn_wq_max
in kamailio-proxy is 32KB by default and that causes an impact
when domain.dump XMLRPC is used on very large domain sets (600+),
as well as sip_domain_reload has improved performance with the removed
XMLRPC domain.dump body parsing.
Change-Id: I17c5718198b06b1ce78b2654f3d7c3bd2830f60b
Rene Krenn
Javier Rodriguez
Kirill Solomko
Oleksandr Duts
Alexander Lutay
Flaviu Mates
Donat Zenichev
Victor Tsvetov
Victor Seva
Sipwise Jenkins Builder
Marco Capetta