bulk-processor

Commit Graph

Author	SHA1	Message	Date
Rene Krenn	efd8299107	MT#61111 escape postgres identifiers Change-Id: I11d6648a4266191759dab63ef3f0b4faa73e1f18	7 months ago
Rene Krenn	2f56063aff	MT#59727 extend cdr data model for cdr exporter support loading the full CDR graph, featuring all relations: -cdr_groups -cdr_tags -cdr_mos -cdr_status -cdr_relations -cdr_presentity -cdr_cash_balances -cdr_time_balances aside various fixes, NGCP::BulkProcessor::Closure is introduced, which provides uniform code execution: -perl code (from YAML) -perl code (from string) -javascript (from string) an identical symbol table is exposed to either language env, including the SQL connector api. Change-Id: If10422df33d996fb6a6a6b53d0ead28ea1cef755	9 months ago
Guillem Jover	bee151585a	TT#53701 Remove executable bit from perl module files Change-Id: I8598dc0ce9382b7b31c946e8f2864fa54c9a92d2	6 years ago
Rene Krenn	8a09b9ab48	TT#37115 normalize line endings Change-Id: Icac176ba667d5fc613654b13a458babb39181f64	7 years ago
Rene Krenn	f8f1e28349	TT#30671 TT#33351 balance free cash restauration tool Change-Id: I738868ae7e8e0e9b665464596aac86ec30856238	7 years ago
Rene Krenn	bfa6575581	TT#18883 implement teletek importer #8 + prepare/merge/clean callforward+voicemail Y/N data + add voip_cf_destinations dao + add voip_cf_destination_sets dao + voip_cf_mappings, destination_sets and destinations insert dao method + write callforwards + write callforward related preference values + not well-formed enough: fixed issues popped up when trying to view/edit created callfowards in panel + fix "database locked" error - properly handle SQLite's serialized transactions (single transaction at a time). + write report file for debugging/review - json graph of merged data of a subscriber - including fields info/warning/error messages for a subscriber - write it snychronized now + get rid of JSON::XS + extend NGCPRestApi connector to support file transfers + heuristic for missing "channels" (by subscriber number count) + prevent mysql deadlocks when writing to ngcp + strictly consider record order from imports + task end result stats polished + cleanup code a bit + add kmailio.location dao + add kmailio.location insert dao method + writing "permanent registrations" to kamailio.location + generate location "ruid" and "partition" according to kamailio Change-Id: Ief9a7634b4930e51d79ac5e963ba48769d3708ea	8 years ago
Rene Krenn	0c667af245	MT#18663 MT#20893 row bulk processing framework WIP #12 +subscriber provisioning via db: +delta handling +create billing subscriber +create provisioning subscriber +create/update billing voip_number +create cli preference +update billing subscriber +create voip_dbalias +create allowed clis preference +release other subscriber's voip numbers +delete obsolete dbalieses +delete obsolete allowed_clis +create kamailio voicemail user +create account_id preference +create cc preference +check config parameters (billing profile of same reseller..) +subscriber and contract termination via api +reprovisioning use-case (upon changed password): -terminate subscriber -terminate contract -create contact, contract, subscriber as normal +log successful provisions +limit to imported batch with --batch +bulk write subscriber preferences directly to db +barring profile to adm_ncos_ID preference +peer_aut_* preferences: init, switchover, clear modes +bulk write via rest-api: +call forwards from features_define.cfg +bulk create lnp_numbers +checks, delta detection for lnp carriers +changed from txn per record (42min) to txn per block (3min) +tuning sqlite params to prevent db lock error with multithreading +workaround for https://rt.cpan.org/Public/Bug/Display.html?id=102791 +include .pl and .cfg file types to be installed by the package +fixup to skip '0' usernames with --skip-errors +check for minimum password length +block rollback issue +import txns only, stmt was not "finished" +fix db_do "E0E" return value +disable multithreading when perldebug +warning counter (--skip-errors only) +fixed locking issue +make the --run=<db-prefix> option work to use different import slite db's Change-Id: I82d27b593b1d5194331c1559b022cf677e232550	9 years ago
Rene Krenn	5e47afdfc9	MT#18663 row bulk processing framework WIP #9 +possibility to load ngcp conf files +db connectivity must be set up explicitly for now.. +start implementing subscriber creation +main record iteration loop, only reading the dump: single thread: ~10min 2 threads: ~5 min 4 threads: ~3.5 min +"check" task +rest-api requests +collection processing +multithreaded processing +rest item object prototype Change-Id: I8ee12037b863f878101e53d69b6ae2b44a96780b	9 years ago
Rene Krenn	1c669a3a73	MT#18663 row bulk processing framework WIP #6 +import LNP_Define.cfg +workingdir folder for rollback logs +cleanup task +chained task execution +dry mode for import +refactoring txn demaraction for rowblock db operations +take out "lock table" impl completely, too dangerous +take out DateTime::Format::Excel dependency for now +allow '#' comments in Subscriber_Define.cfg and Lnp_Define.cfg +check for import sequence +referential integrity checks for Subscriber_Define.cfg +task summary messages / rowcounts +exclude and exclude exception number patterns for Subscriber_Define.cfg +ignore_unique options Change-Id: If1f5094e15828633e212b1a8d651c97816c388b3	9 years ago
Rene Krenn	52d66ca0de	MT#18663 MT#20893 row bulk processing framework WIP #5 +setting to drop duplicate setoptionitems in Feature_Define.cfg +adjust Feature_Define.cfg threads vs block size settings to arrange with a low sqlite busy_timeout +Subscriber_Define.cfg importer +resolve garbage collector problems with log4perl +refactoring to conform with perl module lib/folder structure for installation -runs again Change-Id: I1821cc0085086684c3c1415be4c262453509045a	9 years ago

10 Commits (master)