app_voicemail_odbc: Allow audio to be kept on disk

This commit adds a new voicemail.conf option 'odbc_audio_on_disk'
which when set causes the ODBC variant of app_voicemail to leave
the message and greeting audio files on disk and only store the
message metadata in the database.  This option came from a concern
that the database could grow to large and cause remote access
and/or replication to become slow.  In a clustering situation
with this option, all asterisk instances would share the same
database for the metadata and either use a shared filesystem
or other filesystem replication service much more suitable
for synchronizing files.

The changes to app_voicemail to implement this feature were actually
quite small but due to the complexity of the module, the actual
source code changes were greater.  They fall into the following
categories:

* Tracing.  The module is so complex that it was impossible to
figure out the path taken for various scenarios without the addition
of many SCOPE_ENTER, SCOPE_EXIT and ast_trace statements, even in
code that's not related to the functional change.  Making this worse
was the fact that many "if" statements in this module didn't use
braces.  Since the tracing macros add multiple statements, many "if"
statements had to be converted to use braces.

* Excessive use of PATH_MAX.  Previous maintainers of this module
used PATH_MAX to allocate character arrays for filesystem paths
and SQL statements as though they cost nothing.  In fact, PATH_MAX
is defined as 4096 bytes!  Some functions had (and still have)
multiples of these.  One function has 7.  Given that the vast
majority of installations use the default spool directory path
`/var/spool/asterisk/voicemail`, the actual path length is usually
less than 80 bytes.  That's over 4000 bytes wasted.  It was the
same for SQL statement buffers.  A 4K buffer for statement that
only needed 60 bytes.  All of these PATH_MAX allocations in the
ODBC related code were changed to dynamically allocated buffers.
The rest will have to be addressed separately.

* Bug fixes.  During the development of this feature, several
pre-existing ODBC related bugs were discovered and fixed.  They
had to do with leaving orphaned files on disk, not preserving
original message ids when moving messages between folders,
not honoring the "formats" config parameter in certain circumstances,
etc.

UserNote: This commit adds a new voicemail.conf option
'odbc_audio_on_disk' which when set causes the ODBC variant of
app_voicemail_odbc to leave the message and greeting audio files
on disk and only store the message metadata in the database.
Much more information can be found in the voicemail.conf.sample
file.

(cherry picked from commit 1b3a73cb24)

configs/samples/voicemail.conf.sample

@@ -138,12 +138,93 @@ maxlogins=3
 ;fromstring=The Asterisk PBX
 ; Permit finding entries for forward/compose from the directory
 ;usedirectory=yes
+
+; -----------------------------------------------------------------------------
+; ODBC storage configuration
+; -----------------------------------------------------------------------------
 ; Voicemail can be stored in a database using the ODBC driver.
-; The value of odbcstorage is the database connection configured
-; in res_odbc.conf.
-;odbcstorage=asterisk
-; The default table for ODBC voicemail storage is voicemessages.
-;odbctable=voicemessages
+;
+: Storage database:
+; The value of odbcstorage is the database connection configured in
+; res_odbc.conf.  This may be different from the name of the ODBC DSN
+; in /etc/odbc.ini which, in turn, may be different from the name of the
+; actual database.  If you used the voicemail.ini.sample alembic script
+; located in contrib/ast-db-manage to create the database, the database
+; name is 'voicemail' and the table name is 'voicemail_messages' so you'd
+; need to ensure that /etc/odbc.ini has a DSN entry that points to that
+; database and res_odbc.conf has an entry that points to that ODBC DSN.
+; For historical compatibility, the default value of odbcstorage is
+; actually 'asterisk' because originally voicemail messages were stored
+; in the same database as the rest of the Asterisk configuration.
+
+;odbcstorage = voicemail
+
+; Storage table:
+; The name of the table in which voicemail messages are stored.
+; If you used the voicemail.ini.sample alembic script located in
+; contrib/ast-db-manage to create the database, the table name
+; is 'voicemail_messages'.  For historical compatibility however,
+; the default value of odbctable is 'voicemessages' because originally
+; voicemail messages were stored in that table in the same database
+; as the rest of the Asterisk configuration.
+
+;odbctable = voicemail_messages
+
+; Audio storage location:
+; By default, voicemail and prompt audio files are stored as BLOBs
+; in the database along with the message metadata. If you would
+; prefer to store the audio files on disk and only store the message
+; metadata in the database, set the following option to 'yes'.
+; This can be advantageous in some scenarios, such as when the
+; database is on a separate server from the Asterisk server and
+; network latency and size is a concern or when the database is
+; not well-suited for storing large binary objects.  It can also
+; be useful when you want to use the same voicemail storage
+; configuration for multiple Asterisk servers.  In this situation
+; you can have all the servers use a single shared network file
+; system to store the audio files and use the same database for fast
+; access to the message metadata.
+
+; If you set this option to "yes", new messages and greetings will
+; have their audio kept on disk but the audio for existing messages
+; and greetings will remain in the database until the next time
+; they are played.  At that time, the audio will be moved to disk
+; and erased from the database.
+
+; If you set this option to "no" after you've already stored messages
+; or greetings with it set to "yes", new messages and greetings will
+; have their audio stored in the database but the audio for existing
+; messages and greetings will remain on disk until the next time they
+; are played.  At that time, the audio will be moved to the database
+; and erased from the disk.
+
+; WARNING: Before changing this option from "yes" to "no" or vice
+; versa make sure you have complete backups of your voicemail
+; database and audio files.
+
+; WARNING: If you set this option to "yes" and then later set it
+; to "no", you must ensure that the audio files are not deleted
+; from the disk until you are certain that they have been moved
+; to the database.  If you delete the audio files before they are
+; moved to the database, the messages will be lost.
+
+; WARNING: YOU MUST NOT DOWNGRADE ASTERISK TO A VERSION THAT DOESN'T
+; UNDERSTAND THIS OPTION if you've set this option to "yes" previously
+; and there are audio files stored on disk.  If those files are
+; accessed by a version of Asterisk that doesn't understand this
+; option, the files will be corrupted and the messages will be lost.
+; If you do need to downgrade in this situation, you'll have to write
+; your own script to move the audio files back into the 'recording'
+; column of the database table.  If you record multiple formats,
+; the file to write to the 'recording' column will be the first format
+; listed in the 'format' option in this config file.  If the first
+; format is 'wav49', the file to add to the database will be the one
+; with the 'WAV' extension. 
+
+; odbc_audio_on_disk = no
+
+; -----------------------------------------------------------------------------
+
 ;
 ; Change the from, body and/or subject, variables:
 ;     VM_NAME, VM_DUR, VM_MSGNUM, VM_MAILBOX, VM_CALLERID, VM_CIDNUM,

main/app.c

@@ -1724,7 +1724,11 @@ static int global_maxsilence = 0;
  * \retval 't' Recording ended from the message exceeding the maximum duration, or via DTMF in prepend mode
  * \retval dtmfchar Recording ended via the return value's DTMF character for either cancel or accept.
  */
-static int __ast_play_and_record(struct ast_channel *chan, const char *playfile, const char *recordfile, int maxtime, const char *fmt, int *duration, int *sound_duration, int beep, int silencethreshold, int maxsilence, const char *path, int prepend, const char *acceptdtmf, const char *canceldtmf, int skip_confirmation_sound, enum ast_record_if_exists if_exists)
+static int __ast_play_and_record(struct ast_channel *chan, const char *playfile,
+	const char *recordfile, int maxtime, const char *fmt, int *duration,
+	int *sound_duration, int beep, int silencethreshold, int maxsilence,
+	const char *path, int prepend, const char *acceptdtmf, const char *canceldtmf,
+	int skip_confirmation_sound, enum ast_record_if_exists if_exists)
 {
 	int d = 0;
 	char *fmts;
@@ -1742,6 +1746,8 @@ static int __ast_play_and_record(struct ast_channel *chan, const char *playfile,
 	struct ast_silence_generator *silgen = NULL;
 	char prependfile[PATH_MAX];
 	int ioflags;	/* IO flags for writing output file */
+	SCOPE_ENTER(3, "%s: play: '%s'  record: '%s'  path: '%s'  prepend: %d\n",
+		ast_channel_name(chan), playfile, recordfile, path, prepend);
 
 	ioflags = O_CREAT|O_WRONLY;
 
@@ -1779,19 +1785,22 @@ static int __ast_play_and_record(struct ast_channel *chan, const char *playfile,
 
 	if (playfile || beep) {
 		if (!beep) {
+			ast_trace(-1, "Playing '%s' to '%s'\n", playfile, ast_channel_name(chan));
 			d = ast_play_and_wait(chan, playfile);
 		}
 		if (d > -1) {
+			ast_trace(-1, "Playing 'beep' to '%s'\n", ast_channel_name(chan));
 			d = ast_stream_and_wait(chan, "beep", "");
 		}
 		if (d < 0) {
-			return -1;
+			SCOPE_EXIT_RTN_VALUE(-1, "Failed to play. RC: %d\n", d);
 		}
 	}
 
 	if (prepend) {
 		ast_copy_string(prependfile, recordfile, sizeof(prependfile));
 		strncat(prependfile, "-prepend", sizeof(prependfile) - strlen(prependfile) - 1);
+		ast_trace(-1, "Prepending to '%s'\n", prependfile);
 	}
 
 	fmts = ast_strdupa(fmt);
@@ -1817,7 +1826,7 @@ static int __ast_play_and_record(struct ast_channel *chan, const char *playfile,
 	end = start = time(NULL);  /* pre-initialize end to be same as start in case we never get into loop */
 	for (x = 0; x < fmtcnt; x++) {
 		others[x] = ast_writefile(prepend ? prependfile : recordfile, sfmt[x], comment, ioflags, 0, AST_FILE_MODE);
-		ast_verb(3, "x=%d, open writing:  %s format: %s, %p\n", x, prepend ? prependfile : recordfile, sfmt[x], others[x]);
+		ast_trace(-1, "x=%d, open writing:  %s format: %s, %p\n", x, prepend ? prependfile : recordfile, sfmt[x], others[x]);
 
 		if (!others[x]) {
 			break;
@@ -2108,7 +2117,8 @@ static int __ast_play_and_record(struct ast_channel *chan, const char *playfile,
 			ast_closestream(others[x]);
 			ast_closestream(realfiles[x]);
 			ast_filerename(prependfile, recordfile, sfmt[x]);
-			ast_verb(4, "Recording Format: sfmts=%s, prependfile %s, recordfile %s\n", sfmt[x], prependfile, recordfile);
+			ast_trace(-1, "Recording Format: sfmts=%s, prependfile %s, recordfile %s\n", sfmt[x], prependfile, recordfile);
+			ast_trace(-1, "Deleting the prepend file %s.%s\n", recordfile, sfmt[x]);
 			ast_filedelete(prependfile, sfmt[x]);
 		}
 	} else {
@@ -2130,7 +2140,7 @@ static int __ast_play_and_record(struct ast_channel *chan, const char *playfile,
 	if (sildet) {
 		ast_dsp_free(sildet);
 	}
-	return res;
+	SCOPE_EXIT_RTN_VALUE(res, "Done.  RC: %d\n", res);
 }
 
 static const char default_acceptdtmf[] = "#";