bulk-processor

Commit Graph

Author	SHA1	Message	Date
Rene Krenn	13c1a35d6f	TT#18878 implement Teletek/generic importer #1 - CSV importer: escaped field enclosing, mutliline values, comment char, constant memory - SQLite table to hold flat file content - multithread import for large files - concept of single flatfile abandonend as teletek also wants to import AllowedIPs, MaxConcurrentCalls, NCOS levels, CallForwards - load teletek's main subscriber flatfile - single insert/upsert txn import to identify duplicate cc+ac+sn - alias range unfolding - alias range unfolding fix, analysis of duplicate cc+ac+sn in export_kundinfo_leica2.csv - fixed one-off bug in parser - aliases sorting - aliases dupe check - check for sip account used by different contacts - reseller cache/check for missing reseller - domain cache/check for missing domain - billing profile cache/check for missing billing profile - rest api cert-authentification, multi-user, thread safe, tested - contact create, no update - contract create, no update - billing mapping creaet, no update - contract balance create, no update - billing subscriber, prov subscriber, primary voip_number and voip_dbalias - prefs: account_id, cc, ac - prefs: allowed_cli, cli - load allowed clis file, identify dupes - todo: remove partly adjusted old code parts, depending on this will be extended or not. - add aliases, tested, panel can display 1000+ aliases. - disable multithreading Change-Id: I4f5ae61610bab744e3d0644f7a56afa8874a2c3c	8 years ago
Rene Krenn	5e47afdfc9	MT#18663 row bulk processing framework WIP #9 +possibility to load ngcp conf files +db connectivity must be set up explicitly for now.. +start implementing subscriber creation +main record iteration loop, only reading the dump: single thread: ~10min 2 threads: ~5 min 4 threads: ~3.5 min +"check" task +rest-api requests +collection processing +multithreaded processing +rest item object prototype Change-Id: I8ee12037b863f878101e53d69b6ae2b44a96780b	10 years ago
Rene Krenn	52d66ca0de	MT#18663 MT#20893 row bulk processing framework WIP #5 +setting to drop duplicate setoptionitems in Feature_Define.cfg +adjust Feature_Define.cfg threads vs block size settings to arrange with a low sqlite busy_timeout +Subscriber_Define.cfg importer +resolve garbage collector problems with log4perl +refactoring to conform with perl module lib/folder structure for installation -runs again Change-Id: I1821cc0085086684c3c1415be4c262453509045a	10 years ago

Author

SHA1

Message

Date

Rene Krenn

13c1a35d6f

TT#18878 implement Teletek/generic importer #1

- CSV importer: escaped field enclosing, mutliline values,
  comment char, constant memory
- SQLite table to hold flat file content
- multithread import for large files
- concept of single flatfile abandonend as teletek also wants
  to import AllowedIPs, MaxConcurrentCalls, NCOS levels,
  CallForwards
- load teletek's main subscriber flatfile
- single insert/upsert txn import to identify duplicate
  cc+ac+sn
- alias range unfolding
- alias range unfolding fix, analysis of duplicate cc+ac+sn
  in export_kundinfo_leica2.csv
- fixed one-off bug in parser
- aliases sorting
- aliases dupe check
- check for sip account used by different contacts
- reseller cache/check for missing reseller
- domain cache/check for missing domain
- billing profile cache/check for missing billing profile
- rest api cert-authentification, multi-user, thread safe, tested
- contact create, no update
- contract create, no update
- billing mapping creaet, no update
- contract balance create, no update
- billing subscriber, prov subscriber, primary voip_number and voip_dbalias
- prefs: account_id, cc, ac
- prefs: allowed_cli, cli
- load allowed clis file, identify dupes
- todo: remove partly adjusted old code parts, depending on this will be extended or not.
- add aliases, tested, panel can display 1000+ aliases.

- disable multithreading

Change-Id: I4f5ae61610bab744e3d0644f7a56afa8874a2c3c

Rene Krenn

5e47afdfc9

MT#18663 row bulk processing framework WIP #9

+possibility to load ngcp conf files
 +db connectivity must be set up explicitly for now..
+start implementing subscriber creation
 +main record iteration loop, only reading the dump:
   single thread: ~10min
   2 threads: ~5 min
   4 threads: ~3.5 min
+"check" task
+rest-api requests
 +collection processing
 +multithreaded processing
 +rest item object prototype

Change-Id: I8ee12037b863f878101e53d69b6ae2b44a96780b

Rene Krenn

52d66ca0de

MT#18663 MT#20893 row bulk processing framework WIP #5

+setting to drop duplicate setoptionitems in
 Feature_Define.cfg
+adjust Feature_Define.cfg threads vs block size
 settings to arrange with a low sqlite busy_timeout
+Subscriber_Define.cfg importer
+resolve garbage collector problems with
 log4perl
+refactoring to conform with perl module
 lib/folder structure for installation
 -runs again

Change-Id: I1821cc0085086684c3c1415be4c262453509045a

3 Commits (075cde4c596614eef63af4110ff18ddb4277edeb)