47 Commits (master)

Author	SHA1	Message	Date
Rene Krenn	82af5cf518	MT#62283 "for update no lock" logic for GET /subscribers and /contracts ... to unblock the web UI on heavy loaded systems with call rating enabled, "select .. for update no lock" will no longer wait for acquiring locks when retrieving subscriber or contract contract pages. the contract_balance record creation is will be skipped for such contracts that are locked elsewhere. for rails such as /customerbalances, an explicit contract id row lock timeout is introduced, so it does not depend on the mariadb wait_timeout. Change-Id: I6e96342f3f761279a5876c871d2477977823a39e	2 months ago
Kirill Solomko	205b27a267	MT#59478 API better transaction and error handling ... * Role/Entities*: POST/PUT/PATCH/DELETE methods changes: - support deadlock detection and transaction retry (2 retry attempts at the moment) - improve transaction control, use local $guard instead of saving the ref to $c->stash, as in that case it went out of scope too late and also reported an error message into the log about abnormal $guard out of scope interruption - move all non transaction related code outside of the scope - add error handling when methods such as update_item, and a like do not return the expected data, instead of simply going out of scope and resulting in an uncontrolled reply * Role/API: - rework transaction control: + get_transaction_control() is renamed to start_transaction() to better reflect what it does + complete_transaction() is renamed to commit_transaction() + remove unused %params arg + pass $guard into commit_transaction() instead of having it stored as $c->stash->{transaction_guard) that caused the $guard ref to be destroyed much late than expected (there was also a typo as transaction_quard, which is not relevant anymore with the changes + add check_deadlock() that is invoked when an exception is caught or an $c->errors contain an error, and if the error message represents a transaction error, the transaction block is re-invoked via "goto TX_START" - rework error(): + it now accepts args as following: ($self, $c, $code, $message, @errors) # code -> returned as HTTP code in the reply # message -> returned as HTTP message in the reply # errors -> contain errors for internal logging, last element often contains a DBIx exception + populates all @errors into $c->error so they are available on demend in the code via $c->error or $c->last_error + $c->log->error is not invoked now as the errors become printed in log_response() - log_response() now prints collected errors from $c->error correctly as a separate log line, that is alike to the other api logs so that those can be looked up by the request's tx_id, also all errors are now printed only into api.log * Adjust all $self->error() calls in catch($e) to include $e as the last argument, as well as the duplicate $c->log->error is removed from those ocassions * Remove all $c->log->error() calls as they are replaced with either $self->error() (that logs it correctly into api.log) or $c->error('err') that also adds it correctly into api.log * API::CallForwards: rework to use Entities/EntitiesItem * API::Contracts: rework POST to use Entities * API::PeeringGroups: rework POST to use Entities * API::SubscriberRegistrations: rework POST to use Entities * API::RewriteRuleSets: improve create_item() functionality * Utils/Message: add 'api_retry' log type * $c->session->{api_request_tx_id} is changed to $c->stash->{api_request_tx_id} because sometimes the session ref is different and a different tx_id becomes used Change-Id: I633ce7a8047b1bf00a2f6889003088edf0825dcd	1 year ago
Rene Krenn	0bdd78e230	MT#58422 fix persisting "don't change" topup lock level ... while the api is strict with expecting NULL values, we now tolerate '' (empty string) for topup and underrun lock level values. Change-Id: I77357f0d1152b5af43e39389f8d4d4f3e0b81b2a	2 years ago
Rene Krenn	c73b4e3865	MT#58422 fix persisting "don't change" underrun lock level ... Change-Id: I9d9d99e81facb3548fae81f33349952f4585649c	2 years ago
Kirill Solomko	68af361060	TT#173700 /api/topuplogs expand related fixups ... * Role::API::Contracts add item_by_id() that calls contract_by_id() * Utils::ProfilePackages::catchup_contract_balances return if $contract object is undef * Form::Topup::Log::contract_id expands into Role::API::Customers instead of Role::API::Contracts Change-Id: Id4fc67b8ea1e91f350d0172aafc2b722f34e61f3	3 years ago
Rene Krenn	e3b5ea8860	TT#108250 js warning unknown parameter 'free_time_interval' ... Change-Id: Id4e087626ddf60176556e1384dd967a601811b30	4 years ago
Rene Krenn	a11f2bed5e	TT#77452 properly synchronize subscriber create/update ... acquire the billing.contract row lock *before* any unordered billing.voip_numbers rowlocks by sub manage_subscriber_numbers(). - "deadlock" waittimeout errors will cease when creating subscribers concurrently via api - max_subscribers, is_pilot and other per-contract constraints will be respected accurately Change-Id: I73bb7525b327bbb09217b790be9c14cc65ddebcc	5 years ago
Rene Krenn	6e1b17b26e	TT#44960 /subscribers/?alias=123 w/o group_concat() ... Change-Id: If9466bc4cd6f88cb787c0a707cc32714906dc308	7 years ago
Rene Krenn	7d045b9ffc	TT#42260 remove prepaid and prepaid_library pref usage ... Change-Id: Iebd75291f0648ea054b82ace4251b1a98e629198	7 years ago
Rene Krenn	145167e3db	TT#38401 switch to new billing mappings tables ... Change-Id: I4e57b50a5668170c66521ed5f130bd8293b76b38	7 years ago
Gerhard Jungwirth	6e8dd1a2a1	TT#37221 fix perlcritic error Subroutines::ProhibitExplicitReturnUndef ... this is to clean the Sipwise TAP tests for this repo the critical places where undef is really the intended return value (because a scalar should always be returned) the rule has been overridden Change-Id: I66e217a03a00e0f366db9c0cbc5007dd15cd9f96	7 years ago
Rene Krenn	b35b0e1cbc	TT#35662 move product_id to billing.contracts ... Change-Id: I5381b36e226b1c0c2032fa40c3a082b85485e4ac	7 years ago
Rene Krenn	778c484a5c	TT#34013 altered timezone to get into effect in balances by re-saving contract ... Change-Id: I2e5ee4460115bd4472ff4977e3ad88f1562f2316	7 years ago
Rene Krenn	cee33486f6	TT#34013 save contract_balances in the contract's tz ... Change-Id: I23b4dc82ba8f0a3e535d6c43a195d97704fe69a5	7 years ago
Rene Krenn	c81ca69b70	TT#32910 TT#33037 initial cash_balance, free_time_balance ... + create topup_log records for edit balance operations Change-Id: I6cecbf6025111198b8c43a33b0ffd0f0e30955ad	7 years ago
Rene Krenn	aad8bc802f	TT#29754 debug infos free* ratio caclulation ... Change-Id: Ifa1d59cd2203f5deb00491d91d6ef7683fb1c975	7 years ago
Rene Krenn	57c6cf58ea	TT#21871 add "ratio" field to /api/customerbalances ... + to be used by ngcp-credit-warning script exclusively. Change-Id: Ia549e5dab2ef729b56a521c2793b2dd5a74474e2	8 years ago
Michael Prokop	12234e584d	TT#22411 Fix spelling mistakes/typos ... s/Diffrent/Different/ s/If it mathes/If it matches/ s/addreses/addresses/ s/altough/although/ s/autenticate/authenticate/ s/curently/currently/ s/dont't/don't/ s/dont/don't/ s/existance/existence/ s/explicitely/explicitly/ s/in in/in/ s/lenght/length/ s/neccessary/necessary/ s/occured/occurred/ s/occurence/occurrence/ s/packge/package/ s/paramters/parameters/ s/perfomed/performed/ s/propery/property/ s/recieve/receive/ s/reponse/response/ s/sematical/semantical/ s/serie /series / s/than than/than/ s/the the/the/ s/underun/underrun/ s/wether/whether/ s/wihout/without/ NOTE: Ignoring stuff in share/static + debian/changelog though Change-Id: I90fe718c92582eedf9325d2df3c3280f5f25afd7	8 years ago
Rene Krenn	a0b757a74f	TT#12311 make DateTime add() skip leap seconds ... Change-Id: I267dd6030cb22be62a38b341ae3aade704b9b363	8 years ago
Gerhard Jungwirth	9d2bbdaa76	MT#19509 fix datetime filtering with datatables ... the uppermost level in @searchColumns is to be comined via AND, the next level, via OR (implicitly by DBIx::Class) - plus fix billing networks in customers to be date-searched Change-Id: I8db95cc5e8aadf5d5c86ef8fe06e59eb5e95c3d0	9 years ago
Rene Krenn	9d5085a8ba	MT#19015 correctly use notopup_interval for periodic balance intervals ... +creation of profile packages with hour and minute intervals was disabled in api-balanceintervals.t Change-Id: Idd18ced9704522bf5d919155d866c91b81526b92	9 years ago
Rene Krenn	3b044379b0	MT#18699 control if datatable columns can be sorted ... +disable sorting for profile package's 'Used' count columns Change-Id: I9b9fe28591fb0ee8b22d7b95ce6b527961eab940	9 years ago
Irina Peshinskaya	11256183bd	MT#16321 The smallest changes for particular billing_mappings speed ... Just against ticket issue: subscribers API Change-Id: I77956ac0d0813437824fa83132d1400eba3d2196	10 years ago
Rene Krenn	e5772e77c5	MT#16655 "minute" unit in profile packages ... Change-Id: I5d86459c4d6a1972d11750bebe184f70cdb3472b	10 years ago
Rene Krenn	0b607e5b57	MT#16605 topup log records not persisted for subscribers with lock level ... +failed txn due to to binding a row object instead of value +remove warning with topup voucher validation +remove warning when preparing topup log row data Change-Id: I3a08b6ef297f9b197f30731d886e9d07686336dd	10 years ago
Rene Krenn	997bdc0085	MT#16309 fix deaf logic due to is_create_next typo ... Change-Id: I24ed2b661426fce1b3e52dce7225d4bd44390510	10 years ago
Rene Krenn	64ee5ec1dd	MT#16309 correctly apply changed carry_over modes ... ...with transitions of packages using different start modes Change-Id: I9dfbc2ed5304abc78b632939fcd37858a0352d56	10 years ago
Rene Krenn	1297bdf972	MT#16309 supress notopup balance discard when switching packages ... Change-Id: I8f8c2ab10a4dd32cf5f008e2600d927dd4a977cc	10 years ago
Rene Krenn	fa2c02ee5f	MT#16045 notopup expiry for balance intervals output ... including "timely" topup periods. +for /api/balanceintervals +and for panel UI, "Contract Balance" contract details section reworked api_dump.pl was simplified accordingly. Change-Id: Ib0c4942b1db059b4fe8ee3343838f516662ca166	10 years ago
Rene Krenn	59ec368233	MT#16109 update "prepaid" pref correctly ... Change-Id: I6285db0776a87e4537961fb8a79430d093016b45	10 years ago
Rene Krenn	0009520a84	MT#15759 fix contract balance catchup DST problem ... +basically, the fix means to set billing.contract_balances.start billing.contract_balances.end billing.contracts.create_timestamp billing.contracts.modify_timestamp to the 'local' timezone. +test cases added Change-Id: I203f8fc93df2b53c1cfe89bb1a90cdf6388e20b3	10 years ago
Rene Krenn	8545089145	MT#15399 allow editing profiles, networks, packages ... Change-Id: Iaa989ee18d10a3ea7e871ae99d78d6a2e1e9d40a	10 years ago
Rene Krenn	7f7e3332d1	MT#15477 topup log ... any *attempt* of a top-up request should be logged. +create top-up log records for api in order to record failed top-up requests as well, the somewhat tricky thing is to have two separate transactions here +propery casting of topup request json field values to numbers etc., so the transaction for creating the log entry cannot fail and e.g. requests with subscriber_id='blah' are recorded correctly +new "request_token" parameter for /api/topupvoucher and /api/topupcash, to identify and filter for particular requests. +topup log api resource +topup log api tests +fix to correctly reject used vouchers +topup log panel UI +fix for balanceintervals.t threaded tests Change-Id: I86eb845f6173803705b12cc7e5cdbac9a3153a0a	10 years ago
Rene Krenn	4272b25eee	MT#15477 balance intervals UI ... Change-Id: Ia6abd434870370e08fc7b136e198cbd6d57def2e	10 years ago
Rene Krenn	8494b5a91f	MT#14255 consider balance discard with infinite-end intervals ... +refactoring recent failing api-balanceintervals.t test to need only two concurrent transactions (instead of three, see https://bugtracker.sipwise.com/view.php?id=15277 Change-Id: I4902e616c3adfae81676194fa5e2b99e4b516ed1	10 years ago
Rene Krenn	33e5f4e71f	MT#14255 initial balance_intervals catchup when starting at 1st of month ... Change-Id: I9f22ffc112386da1bdac5e3fedb5b69e4a09fcfa	10 years ago
Rene Krenn	25f54603fb	MT#14255 notopup_expiry calculation only when required ... Change-Id: I79acf1acea6174e4d9f72fccfa01ad9196833fd4	10 years ago
Rene Krenn	07c9d1955d	MT#14255 add cash_balance underrun logic ... in short: whenever the contract_balances "cash_balance" is about to decrease, it might drop below thresholds defined by a contract's profile package. in that case, subscribers are to be locked and/or underrun billing profile sets are to be applied. this commit also contains some minor fixes: +when creating/editing profile packages, all billing profiles of initial, topup, underrun sets must show the same interval_free_cash and interval_free_time values. there was also a problem with adding profile sets to packages that use profile with prepaid = 1 (see ticket 14301) +when editing packages using the panel UI, the validation was not properly invoked (!) +the frequency of topups has to be limited to 1 per second. +updating the "lock" subscriber prior to editing or viewing it in the panel UI was added. +topup cash and topup voucher panel UI added +when aquiring multiple contract row locks at once, no joined tables are used any mure. the number of row locks is therefore minimal now (see ticket 14903) +rework of updating prepaid subscriber preference Change-Id: Ideb0fec2404ca10b3ac5d8735f53620d0c3a24ed	10 years ago
Rene Krenn	c7b3d74a21	MT#14317 passing now() to interval catchup and resize ... formerly, the contract_balance catchup (and resize) procedures used contracts.modified_timestamp, whose value was intended to be set explicitly upon update operations. it turns out that mysql ON UPDATE current_timestamp columns effectively ignore the explicit timestamp value, unless it exactly matches the old timestamp value. while this is not an issue for the logic in production usage, this seems to be the reason why the api-balance_intervals.t test fails occassionally with its time-warp cases. Change-Id: I504dd98548b88c99079e071598806d884c4422eb	10 years ago
Rene Krenn	ff75e75138	MT#14317 add debug info of panel interval catchups ... Change-Id: I1d37ec624c6642f54187ba463cf1121df4eb19a4	10 years ago
Rene Krenn	bef06dce91	MT#14301 deleting prof packages instead of terminating ... +profile package details view to overlook associations +voucher count is properly checked before deleting a package +package_id filters for /api/customers and /api/vouchers +Fake::Time replaced by Time::Warp for balance_interval tests +cleanup Change-Id: Idc34220a9d29c115453eeee2907e70a71cf9f0cc	10 years ago
Rene Krenn	ca908406a1	MT#13903 enable applying profile package ... Change-Id: I74b57b80599575e9685e01924b234c8da6c2fe49	10 years ago
Rene Krenn	f3b711a050	MT#13903 disable applying profile package ... This is a patcht introducing a variable to disable setting profile_package_id's in in the contracts table. The code should therefore behave as before, as long rate-o-mat and libswrate are ready. Not once set to 1 and contract balances are written to a database once, disabling this constant will mess up contract_balances! Change-Id: I080460395627202f3f5a386022de643577f4f3c2	10 years ago
Rene Krenn	d54bd5d360	MT#13903 "topup_interval" start mode impl ... The prior "topup" start mode for balance intervals (contract_balances) creates intervals for each top-up using a voucher, resulting in dynamic length. This new "topup_interval" start mode will just align the *first* balance interval upon the topup. Subsequent intervals use the constant interval unit+count specified in the profile package. Melita user story is completely covered now. Change-Id: I0a8b8d73a7fbf054d0aa8b2b1e73a54609fdeee0	10 years ago
Rene Krenn	bd9f67040d	MT#13903 topupvoucher and topupcash ... +applying profile package and billing mappings +testcase in api-balanceintervals.t +caveats: to meet melita's user story, an additional 'topup_interval' interval start mode will be required. the currently implemented 'topup' start mode restarts intervals upon every topup and therefore does not provide constantinterval lengths. Change-Id: I0a4898783c023749994e94e6909833a42debe259	10 years ago
Rene Krenn	c378681a24	MT#13903 balance interval catchup and resize WIP ... + synchronized contract balance catchup + balance interval resizing upon profile package transitions + dynamic interval length, interval start, carry-over propagation, .. + end-of-month 'preserve' mode correction for strictly aligned month intervals for start_mode=customer "create" timestamp + new api/balanceintervals resource to inspect contracts' balance interval histories + updated affected panel UI controllers + test case with time warps using Time::Fake caveats: - creating an invoice for a given 'period' (month) has to be refactored to select a disitnct balance interval. right now it takes the last interval in the month specified. - generate_invoice.pl is broken and needs a major refactoring Change-Id: I7bb54a83b76e510b1baa573a986d05400a7fec1e	10 years ago
Rene Krenn	2537819f0d	MT#12643 billing networks and multi bill prof WIP ... -properly validate billing mapping intervals -api/contracts and api/customers complete +the problem with updating the "prepaid" preference remains at the moment +billing_mappings field contains future (editable) mappings only, the full schedule is given in a separate field +hal-links to all occurring billing profiles and networks -extended tests: api-customer.t and api-contract.t -api-billingnetworks.t test -billing networks panel controller +search by enclosed IP address -panel UIs for billing networks, customers, contracts +past billing mapping intervals that cannot be edited are displayed by a seperate hfh "block" part of the billing_profiles repeatable -profile packages api, UI +terminating billing profiles (MT#8645) will be broken +fixed now: profile cannot be terminated unless there are: -zero non-terminated customers, whose mappings contain the profile with future or open end -zero profile packages, that link to the profile in either one of their initial-, underrun- or topup profile sets +panel edit should work, termination wip +topup_profiles w/o amounts +contracts/customers API profile_package_id field: +apply the package's initial profiles +api/customer and api/contract profile definition modes (id/profiles/package) +api OK, api-customer.t tests +customer and contract panel UI with selector to specify actual billing mapping, schedule or profile package (same as api) +terminate logic harmonized: profiles, networks, packages +api tests updated +prepare for merging: cleanup, enable tests Change-Id: I6c0dec72f490b5731fc673cd393d6aeb08dd8143	10 years ago