tool to switch contracts with some billing profile
to a desired profile package, as specified in mappings.yml.
run like:
perl process.pl --task=cleanup --task=set_profile_package --skip-errors
Change-Id: Ie7c07642ed29d91166aad95dfb62a0e3e40a5e3f
(cherry picked from commit 2bd51c65c1)
+ introduce support to manipulate contract, domain, etc.
voip_preferences
+ write concurrent_max_total preference
+ error in invalid "channels = 0"
+ try to find non-empty sip-password of a subscriber, if any
+ try to find nonempty web username+password, if any
+ set emtpy web_password, if web_username is empty
+ auto-generate web_username, if duplicate
+ auto-generate web_password, if web_username is not emtpy but
web_password is empty
+ parse export_CLIR.csv
+ write clir preference
+ write allowed_ips_grp preference
+ use maximum of concurrent_max_total for a subscriber,
when inconsisten
+ use combined list of allowed ips, when inconsistent
+ write adm_ncos, derive barrings combination if
inconsistent for a subscriber
+ log created items, cleanup code a bit
+ range unfolding for numbers in allowedcli files
Change-Id: I26bab2cb9b4ef349777ebef2e883c9e41af3eeba
+ "allowed_cli" import gets a different meaning,
it are additional "numbers" (alias+allowed_cli) to import
+ unique cc+ac+sn in subscriber file
+ unique cc+ac+sn in allowed_cli file
+ logging of cc+ac+sn of allowed_cli records
already found in subscriber file
+ logging of sip_usernames of allowed_cli records
not found in subscriber file
+ reseller mapping
Change-Id: I620db415f5dab91e8d5c6336b7a45b3a0ffeacb9
+ allow to specify multiple subscriber files to import
+ allow to specify multiple allowed_cli files to import
Change-Id: I80f83542f541a0d8857d477158dc043cfabb93ba
- CSV importer: escaped field enclosing, mutliline values,
comment char, constant memory
- SQLite table to hold flat file content
- multithread import for large files
- concept of single flatfile abandonend as teletek also wants
to import AllowedIPs, MaxConcurrentCalls, NCOS levels,
CallForwards
- load teletek's main subscriber flatfile
- single insert/upsert txn import to identify duplicate
cc+ac+sn
- alias range unfolding
- alias range unfolding fix, analysis of duplicate cc+ac+sn
in export_kundinfo_leica2.csv
- fixed one-off bug in parser
- aliases sorting
- aliases dupe check
- check for sip account used by different contacts
- reseller cache/check for missing reseller
- domain cache/check for missing domain
- billing profile cache/check for missing billing profile
- rest api cert-authentification, multi-user, thread safe, tested
- contact create, no update
- contract create, no update
- billing mapping creaet, no update
- contract balance create, no update
- billing subscriber, prov subscriber, primary voip_number and voip_dbalias
- prefs: account_id, cc, ac
- prefs: allowed_cli, cli
- load allowed clis file, identify dupes
- todo: remove partly adjusted old code parts, depending on this will be extended or not.
- add aliases, tested, panel can display 1000+ aliases.
- disable multithreading
Change-Id: I4f5ae61610bab744e3d0644f7a56afa8874a2c3c
The latter does not support YAML 1.1, nor many parts of the
specification. Use the more compliant implementation, in addition to try
to converge to a single one, so that we do not get serialization delta
surprises.
Change-Id: Ia3f6808b9bb84afb224cc29040964eefe07bf118
fix tool to update acc_trash and re-enqueue records accumulated
during 3.x->4.x updates due to introduction of cdr lnp prefix
fields.
Change-Id: Iea6140edccc1aeb456f6584b7d8335e18ba1b69d
+initial process.pl
+contract balance gap fix thread
+contract iteration
+context init
+context teardown
+detectig gaps and overlaps
+insert missing contract balances:
no packages or free time considered for now
+calendar implementation
+drop URI::Encode dependency
+for now, adopt to run with mr3.8 tables only.
Change-Id: I48e0de59bb703adc0d28092c1a93a5b47432bc03
Build-Depend on libmodule-build-perl which is not part of the perl
distribution anymore, or the perl versions that provided the module.
Change-Id: Ie62c0b0a39778dc010ef5801577f9e3537d77810
+importing lnp numbers now requires unique numbers,
it required unique prefix+number before
+ngcp billing.lnp_number records are considered as
unique per number (before it was unique per
prefix+number)
Change-Id: I5a6f9ed4235a1ee0edecd58ec3b04ad393839dbb
+tolerant line endings for the username+password import
+same for batch files
+only process batch records with updated or added delta
+convenience: no need to clear the batch import before
loading a new batch file.
+if records in a batch file overlap with ones from previous
batch files, they are "updated" and hece processed again.
+subscriber password update instead of recreating subs
(aka know as reprovisioning).
+if there's a peer_auth_pass present, its also updated.
Change-Id: I22ea6485dd2d48e3633ebb262037c8d3fafc923c
+subscriber provisioning via db:
+delta handling
+create billing subscriber
+create provisioning subscriber
+create/update billing voip_number
+create cli preference
+update billing subscriber
+create voip_dbalias
+create allowed clis preference
+release other subscriber's voip numbers
+delete obsolete dbalieses
+delete obsolete allowed_clis
+create kamailio voicemail user
+create account_id preference
+create cc preference
+check config parameters (billing profile of same reseller..)
+subscriber and contract termination via api
+reprovisioning use-case (upon changed password):
-terminate subscriber
-terminate contract
-create contact, contract, subscriber as normal
+log successful provisions
+limit to imported batch with --batch
+bulk write subscriber preferences directly to db
+barring profile to adm_ncos_ID preference
+peer_aut_* preferences: init, switchover, clear modes
+bulk write via rest-api:
+call forwards from features_define.cfg
+bulk create lnp_numbers
+checks, delta detection for lnp carriers
+changed from txn per record (42min) to txn per block (3min)
+tuning sqlite params to prevent db lock error
with multithreading
+workaround for
https://rt.cpan.org/Public/Bug/Display.html?id=102791
+include .pl and .cfg file types to be installed by the
package
+fixup to skip '0' usernames with --skip-errors
+check for minimum password length
+block rollback issue
+import txns only, stmt was not "finished"
+fix db_do "E0E" return value
+disable multithreading when perldebug
+warning counter (--skip-errors only)
+fixed locking issue
+make the --run=<db-prefix> option work to use
different import slite db's
Change-Id: I82d27b593b1d5194331c1559b022cf677e232550
+provisioning db connectivity
+kamailio db connectivity
+default db connectivity for distributed txn's (XA)
+provisioning db dao's
+kamailio db dao's
+subscriber provisioning via db:
+create contact
+create contract
+create billing_mapping
+create contract_balance
Change-Id: I11de85fa5473e59bd6a8c05e1541a9fbc96c1a9d
+possibility to load ngcp conf files
+db connectivity must be set up explicitly for now..
+start implementing subscriber creation
+main record iteration loop, only reading the dump:
single thread: ~10min
2 threads: ~5 min
4 threads: ~3.5 min
+"check" task
+rest-api requests
+collection processing
+multithreaded processing
+rest item object prototype
Change-Id: I8ee12037b863f878101e53d69b6ae2b44a96780b
+delta calculation for lnp,features,featureoptions
+skip prepaid subscribers
+import username+password.txt
-a full file with plaintext instead of hashes
is needed to test..
+import order forced
+import "batch.txt"
-prepared for a single column containing the
numbers to migrate
Change-Id: Icfedda64b635fe872f6f24c005228773d9869b21
+allow #* symbols in setoptionitems in
Features_Define.cfg
+prevent double execution+log properly
+make sure to close all db connections before
forking threads, not only project-specific ones
+per-db dao folder for ipgallery migration project
+start with ngcp db dao's:
+contract_balances
+contracts
+refactor dao/sql processing class
+params hash for method with long arg list
+take out obselete methods
+dedicated unique field option for "insert_record"
+split SqlRecord class into
SqlRecord&SqlProcessor
+dependencies in Build.pl and debian/control
+took out comments from debian/control
+calcualte deltas of Subscriber_Define.cfg imports
+"truncate subscriber import" task
Change-Id: I8eab88319e6a4ba3d5b6ba935e50a56cc1763ad9
+import LNP_Define.cfg
+workingdir folder for rollback logs
+cleanup task
+chained task execution
+dry mode for import
+refactoring txn demaraction for rowblock
db operations
+take out "lock table" impl completely,
too dangerous
+take out DateTime::Format::Excel dependency
for now
+allow '#' comments in Subscriber_Define.cfg
and Lnp_Define.cfg
+check for import sequence
+referential integrity checks for Subscriber_Define.cfg
+task summary messages / rowcounts
+exclude and exclude exception number patterns
for Subscriber_Define.cfg
+ignore_unique options
Change-Id: If1f5094e15828633e212b1a8d651c97816c388b3
+setting to drop duplicate setoptionitems in
Feature_Define.cfg
+adjust Feature_Define.cfg threads vs block size
settings to arrange with a low sqlite busy_timeout
+Subscriber_Define.cfg importer
+resolve garbage collector problems with
log4perl
+refactoring to conform with perl module
lib/folder structure for installation
-runs again
Change-Id: I1821cc0085086684c3c1415be4c262453509045a
+"import" stage continued:
+Dao's/db schema for the sqlite db to map the
Features_Define.cfg content
+project-specific connector pool
+Importing Features_Define.cfg is working,
1.2+4.2 mio. records can be inserted into the temporary
sqlite db in 4min40sec using 4 hyperthread cpus
+importing melita's remaining files from the set will
be less complicated by far
+GetOpt setup
+a stage is a task, multiple tasks can be passed
as args, to be executed at once
+dry mode option
+force option
+project settings file extended
Change-Id: I6a84eb90756ad382dc3e96636220254b5e3b5023
+input/output folder permission
-> process umask fix
+config file loading fallbacks
+refactor to allow custom additions within
"Projects" subdirs
+settings file per project
+make RestConnectors compiling
+make thread states used in thread queue patterns
more verbose/readable
+multithreaded text file processor
-linesplitter, fieldsplitter callbacks
+poc: marpa r2 parsergenerator to parse
Features_Define.cfg by declaring the syntax via BNF
+try to make it work
+evaluating the syntax tree to get a final
row data structure turns out to be slow
+hence moving parsing to processor threads
+evaluating a Features_Define.cfg record into a
perl data structure succeeded.
+class hierarchy for file processors
+main "migrate.pl" layout
Change-Id: I1550e2689bb5640931787fc70e9b5b00432dd0a2
Sipwise Jenkins Builder
Rene Krenn
Guillem Jover
Michael Prokop
Alexander Lutay
Victor Seva