309 Commits (master)

 

All Branches   

Search

Author	SHA1	Message	Date
Rene Krenn	b4f244a9f1	MT#18663 MT#20893 row bulk processing framework WIP #7 ... +allow #* symbols in setoptionitems in Features_Define.cfg +prevent double execution+log properly +make sure to close all db connections before forking threads, not only project-specific ones +per-db dao folder for ipgallery migration project +start with ngcp db dao's: +contract_balances +contracts +refactor dao/sql processing class +params hash for method with long arg list +take out obselete methods +dedicated unique field option for "insert_record" +split SqlRecord class into SqlRecord&SqlProcessor +dependencies in Build.pl and debian/control +took out comments from debian/control +calcualte deltas of Subscriber_Define.cfg imports +"truncate subscriber import" task Change-Id: I8eab88319e6a4ba3d5b6ba935e50a56cc1763ad9	10 years ago
Rene Krenn	1c669a3a73	MT#18663 row bulk processing framework WIP #6 ... +import LNP_Define.cfg +workingdir folder for rollback logs +cleanup task +chained task execution +dry mode for import +refactoring txn demaraction for rowblock db operations +take out "lock table" impl completely, too dangerous +take out DateTime::Format::Excel dependency for now +allow '#' comments in Subscriber_Define.cfg and Lnp_Define.cfg +check for import sequence +referential integrity checks for Subscriber_Define.cfg +task summary messages / rowcounts +exclude and exclude exception number patterns for Subscriber_Define.cfg +ignore_unique options Change-Id: If1f5094e15828633e212b1a8d651c97816c388b3	10 years ago
Rene Krenn	52d66ca0de	MT#18663 MT#20893 row bulk processing framework WIP #5 ... +setting to drop duplicate setoptionitems in Feature_Define.cfg +adjust Feature_Define.cfg threads vs block size settings to arrange with a low sqlite busy_timeout +Subscriber_Define.cfg importer +resolve garbage collector problems with log4perl +refactoring to conform with perl module lib/folder structure for installation -runs again Change-Id: I1821cc0085086684c3c1415be4c262453509045a	10 years ago
Victor Seva	cc3e93d77b	MT#20893 Initial dummy package ... Change-Id: I6d80f83c6cd4ec59959b65a21826a6adf3adf0fe	10 years ago
Rene Krenn	4570a01bbc	MT#18663 row bulk processing framework WIP #4 ... +"import" stage continued: +Dao's/db schema for the sqlite db to map the Features_Define.cfg content +project-specific connector pool +Importing Features_Define.cfg is working, 1.2+4.2 mio. records can be inserted into the temporary sqlite db in 4min40sec using 4 hyperthread cpus +importing melita's remaining files from the set will be less complicated by far +GetOpt setup +a stage is a task, multiple tasks can be passed as args, to be executed at once +dry mode option +force option +project settings file extended Change-Id: I6a84eb90756ad382dc3e96636220254b5e3b5023	10 years ago
Rene Krenn	0d40b0c4ff	MT#18663 row bulk processing framework WIP #3 ... +input/output folder permission -> process umask fix +config file loading fallbacks +refactor to allow custom additions within "Projects" subdirs +settings file per project +make RestConnectors compiling +make thread states used in thread queue patterns more verbose/readable +multithreaded text file processor -linesplitter, fieldsplitter callbacks +poc: marpa r2 parsergenerator to parse Features_Define.cfg by declaring the syntax via BNF +try to make it work +evaluating the syntax tree to get a final row data structure turns out to be slow +hence moving parsing to processor threads +evaluating a Features_Define.cfg record into a perl data structure succeeded. +class hierarchy for file processors +main "migrate.pl" layout Change-Id: I1550e2689bb5640931787fc70e9b5b00432dd0a2	10 years ago
Rene Krenn	656e373f57	MT#18663 row bulk processing framework WIP #2 ... +included Excel::Reader::XLSX, as it was discontinued +FakeTime +rest connectors class hierarchy +ngcp rest api connector +support for yml config files +log4perl initialisation refactored +separate a "working paht" from the application_path, to be friendly with packaging/installation Change-Id: If56c27a3c3b8b526b5abc7a29e049a0db6a43ad6	10 years ago
Rene Krenn	9027e86928	MT#18663 row bulk processing framework WIP ... initial commit .gitignore .percriticrc - doesn't work no critic per file Change-Id: If46efa9a6b008861d3e7a527c47715743fef2579	10 years ago
Alexander Lutay	538fc41bd3	MT#18529 Add .gitreview for bulk-processor ... Change-Id: If65a1ea5427f156f8bae6f037443c40870bfb406	10 years ago