bulk-processor

Commit Graph

Author	SHA1	Message	Date
Rene Krenn	a2d365e3df	MT#18663 MT#20893 row bulk processing framework WIP #14 +support for handling multiple ngcp table versions at runtime +setting allowed_ips Change-Id: I7e87456d561407fd35b4518ed53a3c435c2e7798	9 years ago
Rene Krenn	0332ec4fc3	MT#18663 MT#20893 row bulk processing framework WIP #13 +dedicated '356' prefix for subscriber usernames +add (reimplement) setting allowed_ips +WIP Change-Id: If09b6841b7ed9abf7ea545c1f51ee9c494f40ffe	9 years ago
Rene Krenn	0c667af245	MT#18663 MT#20893 row bulk processing framework WIP #12 +subscriber provisioning via db: +delta handling +create billing subscriber +create provisioning subscriber +create/update billing voip_number +create cli preference +update billing subscriber +create voip_dbalias +create allowed clis preference +release other subscriber's voip numbers +delete obsolete dbalieses +delete obsolete allowed_clis +create kamailio voicemail user +create account_id preference +create cc preference +check config parameters (billing profile of same reseller..) +subscriber and contract termination via api +reprovisioning use-case (upon changed password): -terminate subscriber -terminate contract -create contact, contract, subscriber as normal +log successful provisions +limit to imported batch with --batch +bulk write subscriber preferences directly to db +barring profile to adm_ncos_ID preference +peer_aut_* preferences: init, switchover, clear modes +bulk write via rest-api: +call forwards from features_define.cfg +bulk create lnp_numbers +checks, delta detection for lnp carriers +changed from txn per record (42min) to txn per block (3min) +tuning sqlite params to prevent db lock error with multithreading +workaround for https://rt.cpan.org/Public/Bug/Display.html?id=102791 +include .pl and .cfg file types to be installed by the package +fixup to skip '0' usernames with --skip-errors +check for minimum password length +block rollback issue +import txns only, stmt was not "finished" +fix db_do "E0E" return value +disable multithreading when perldebug +warning counter (--skip-errors only) +fixed locking issue +make the --run=<db-prefix> option work to use different import slite db's Change-Id: I82d27b593b1d5194331c1559b022cf677e232550	9 years ago
Rene Krenn	15762ef304	MT#18663 row bulk processing framework WIP #11 +provisioning db connectivity +kamailio db connectivity +default db connectivity for distributed txn's (XA) +provisioning db dao's +kamailio db dao's +subscriber provisioning via db: +create contact +create contract +create billing_mapping +create contract_balance Change-Id: I11de85fa5473e59bd6a8c05e1541a9fbc96c1a9d	9 years ago
Rene Krenn	84575f90e6	MT#18663 row bulk processing framework WIP #10 +reject invalid subscriber numbers by pattern +detect changed passwords in subscriber+username.xslx +separate --skip-errors option from --dry option +rest request classes for ngcp resources: +SystemContacts +Contracts +BillingProfiles +Resellers +BillingZones +BillingFees +Domains +CustomerContacts +Customers +Subscribers +making post, get, put, patch, delete work for NGCP rest-api Change-Id: Ic55caed7d5425716adfbba9e2bc6b9153334481c	9 years ago
Rene Krenn	5e47afdfc9	MT#18663 row bulk processing framework WIP #9 +possibility to load ngcp conf files +db connectivity must be set up explicitly for now.. +start implementing subscriber creation +main record iteration loop, only reading the dump: single thread: ~10min 2 threads: ~5 min 4 threads: ~3.5 min +"check" task +rest-api requests +collection processing +multithreaded processing +rest item object prototype Change-Id: I8ee12037b863f878101e53d69b6ae2b44a96780b	9 years ago
Rene Krenn	8eece8ecd1	MT#18663 row bulk processing framework WIP #8 +delta calculation for lnp,features,featureoptions +skip prepaid subscribers +import username+password.txt -a full file with plaintext instead of hashes is needed to test.. +import order forced +import "batch.txt" -prepared for a single column containing the numbers to migrate Change-Id: Icfedda64b635fe872f6f24c005228773d9869b21	9 years ago
Rene Krenn	b4f244a9f1	MT#18663 MT#20893 row bulk processing framework WIP #7 +allow #* symbols in setoptionitems in Features_Define.cfg +prevent double execution+log properly +make sure to close all db connections before forking threads, not only project-specific ones +per-db dao folder for ipgallery migration project +start with ngcp db dao's: +contract_balances +contracts +refactor dao/sql processing class +params hash for method with long arg list +take out obselete methods +dedicated unique field option for "insert_record" +split SqlRecord class into SqlRecord&SqlProcessor +dependencies in Build.pl and debian/control +took out comments from debian/control +calcualte deltas of Subscriber_Define.cfg imports +"truncate subscriber import" task Change-Id: I8eab88319e6a4ba3d5b6ba935e50a56cc1763ad9	9 years ago
Rene Krenn	1c669a3a73	MT#18663 row bulk processing framework WIP #6 +import LNP_Define.cfg +workingdir folder for rollback logs +cleanup task +chained task execution +dry mode for import +refactoring txn demaraction for rowblock db operations +take out "lock table" impl completely, too dangerous +take out DateTime::Format::Excel dependency for now +allow '#' comments in Subscriber_Define.cfg and Lnp_Define.cfg +check for import sequence +referential integrity checks for Subscriber_Define.cfg +task summary messages / rowcounts +exclude and exclude exception number patterns for Subscriber_Define.cfg +ignore_unique options Change-Id: If1f5094e15828633e212b1a8d651c97816c388b3	9 years ago
Rene Krenn	52d66ca0de	MT#18663 MT#20893 row bulk processing framework WIP #5 +setting to drop duplicate setoptionitems in Feature_Define.cfg +adjust Feature_Define.cfg threads vs block size settings to arrange with a low sqlite busy_timeout +Subscriber_Define.cfg importer +resolve garbage collector problems with log4perl +refactoring to conform with perl module lib/folder structure for installation -runs again Change-Id: I1821cc0085086684c3c1415be4c262453509045a	9 years ago

10 Commits (a2d365e3df8233c8183677695e87c88e376653cb)