+-----------------------+
                        |    INTERNET / PSTN    |
                        +-----------+-----------+
                                    |
                          SIP Trunks (inbound/outbound)
                                    |
            +-----------------------+-----------------------+
            |                       |                       |
    +-------+-------+     +--------+-------+     +---------+------+
    | DIALER 1      |     | DIALER 2       |     | DIALER 3       |
    | (Asterisk)    |     | (Asterisk)     |     | (Asterisk)     |
    | 10.0.0.11     |     | 10.0.0.12      |     | 10.0.0.13      |
    | Conf: 8600001 |     | Conf: 8600201  |     | Conf: 8600401  |
    |   - 8600200   |     |   - 8600400    |     |   - 8600600    |
    +-------+-------+     +--------+-------+     +---------+------+
            |   \                  |                  /   |
            |    \---- IAX2 ------+------ IAX2 -----/    |
            |         Trunks      |       Trunks         |
            |                     |                      |
    +-------+---------------------+----------------------+-------+
    |                     NETWORK (LAN/VLAN)                     |
    +-------+---------------------+----------------------+-------+
            |                     |                      |
    +-------+-------+    +-------+--------+    +--------+-------+
    | DATABASE      |    | WEB SERVER     |    | WEB SERVER     |
    | (MariaDB)     |    | (Apache+PHP)   |    | (Apache+PHP)   |
    | 10.0.0.1      |    | 10.0.0.2       |    | 10.0.0.3       |
    |               |    | Agent UI       |    | Admin/Reports  |
    | Central hub   |    | Points to DB   |    | Points to DB   |
    | for ALL data  |    | at 10.0.0.1    |    | at 10.0.0.1    |
    +---------------+    +----------------+    +----------------+

CRITICAL NETWORK RULES:

1. All servers MUST be on the same LAN or VLAN
   - Latency between any two nodes: < 5ms (ideally < 1ms)
   - DB queries happen thousands of times per second
   - High latency = slow agent screens, dropped conferences

2. If servers are in different data centers:
   - Use a private VLAN or VPN between them
   - Latency > 20ms will cause noticeable agent UI lag
   - Latency > 50ms will cause conference/transfer failures

3. Firewall rules between cluster nodes:
   - TCP 3306 (MariaDB) - DB server from all nodes
   - TCP 80/443 (Apache) - Web server from agents/admins
   - UDP 4569 (IAX2) - Between all dialer servers
   - UDP 5060 + 10000-20000 (SIP + RTP) - Dialer to SIP trunks
   - TCP 4577 (FastAGI) - Loopback on each dialer (127.0.0.1 only)
   - ICMP - Allow between all nodes for health checks

4. All servers MUST have static IPs
   - ViciDial uses IP addresses as server identifiers
   - Changing an IP requires reconfiguring the entire cluster

# Install chrony (preferred over ntpd)
# openSUSE / ViciBox:
zypper install chrony

# AlmaLinux / Rocky:
dnf install chrony

# Upstream NTP servers
pool pool.ntp.org iburst maxsources 4
pool time.google.com iburst maxsources 2

# Allow cluster nodes to sync from this server
allow 10.0.0.0/24

# Serve time even when not synchronized to upstream
local stratum 10

# Log statistics
logdir /var/log/chrony
log measurements statistics tracking

systemctl enable chronyd
systemctl start chronyd

# Verify
chronyc sources -v
chronyc tracking

# Sync ONLY from the DB server (cluster NTP master)
server 10.0.0.1 iburst prefer

# Fallback to public pool if master is down
pool pool.ntp.org iburst maxsources 2

# Step the clock if drift exceeds 1 second (important for first sync)
makestep 1.0 3

logdir /var/log/chrony
log measurements statistics tracking

systemctl enable chronyd
systemctl restart chronyd

# Verify sync to master
chronyc sources -v
# Should show 10.0.0.1 as preferred source with * indicator

# Run on each server -- times should be within 0.01s of each other
date +%s.%N

10.0.0.1    vici-db      vici-db.local
10.0.0.2    vici-web1    vici-web1.local
10.0.0.11   vici-dial1   vici-dial1.local
10.0.0.12   vici-dial2   vici-dial2.local
10.0.0.13   vici-dial3   vici-dial3.local

hostnamectl set-hostname vici-db      # On DB server
hostnamectl set-hostname vici-dial1   # On Dialer 1
# etc.

# Install SVN client
# openSUSE:
zypper install subversion

# AlmaLinux:
dnf install subversion

# Check out the ViciDial source (trunk = latest)
mkdir -p /usr/src/astguiclient
cd /usr/src/astguiclient
svn checkout svn://svn.eflo.net/agc_2-X/trunk

# The source tree will be at /usr/src/astguiclient/trunk/

zypper install mariadb mariadb-client
systemctl enable mariadb
systemctl start mariadb

dnf install mariadb-server mariadb
systemctl enable mariadb
systemctl start mariadb
mysql_secure_installation

# ViciBox / openSUSE: /etc/my.cnf
# AlmaLinux: /etc/my.cnf.d/mariadb-server.cnf

[mysqld]
# CRITICAL: Bind to all interfaces (or specific LAN IP)
bind-address = 0.0.0.0
# Alternative: bind to specific LAN IP only (more secure)
# bind-address = 10.0.0.1

# Do NOT have skip-networking enabled -- remove or comment it out
# skip-networking

# Server ID (required, unique per server, useful if adding replication later)
server-id = 1

# Port (default 3306)
port = 3306

# Character set
character-set-server = utf8
collation-server = utf8_unicode_ci

# InnoDB settings (tune based on available RAM)
# Set to 60-70% of total RAM on a dedicated DB server
innodb_buffer_pool_size = 20G
innodb_log_file_size = 512M
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_file_per_table = 1
innodb_io_capacity = 2000
innodb_io_capacity_max = 4000

# Connection limits (must handle all cluster nodes)
max_connections = 500
max_connect_errors = 100000

# Query safety -- kill long-running queries from web/cron
max_statement_time = 300

# Timeouts
wait_timeout = 28800
interactive_timeout = 28800
net_read_timeout = 60
net_write_timeout = 120

# Temp tables
tmp_table_size = 256M
max_heap_table_size = 256M

# Query cache (MariaDB 10.x)
query_cache_type = 1
query_cache_size = 128M

# Thread handling
thread_cache_size = 64
table_open_cache = 4096

# Logging
log-error = /var/log/mysql/mysqld.log
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 5

# Binary logging (recommended for backups and future replication)
log_bin = mysql-bin
binlog_format = MIXED
expire_logs_days = 7

systemctl restart mariadb

ss -tlnp | grep 3306
# Should show: 0.0.0.0:3306 or 10.0.0.1:3306

cd /usr/src/astguiclient/trunk

# Run the install script, it creates the 'asterisk' database and all tables
perl install.pl

mysql -u root -p

-- ============================================================
-- GRANT STATEMENTS FOR VICIDIAL CLUSTER
-- Replace passwords with strong, unique values!
-- ============================================================

-- User: cron (primary ViciDial operations user)
-- Needs full access to the 'asterisk' database from each server

-- From Web Server 1 (10.0.0.2)
CREATE USER 'cron'@'10.0.0.2' IDENTIFIED BY 'Str0ng_Cr0n_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'cron'@'10.0.0.2';

-- From Dialer 1 (10.0.0.11)
CREATE USER 'cron'@'10.0.0.11' IDENTIFIED BY 'Str0ng_Cr0n_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'cron'@'10.0.0.11';

-- From Dialer 2 (10.0.0.12)
CREATE USER 'cron'@'10.0.0.12' IDENTIFIED BY 'Str0ng_Cr0n_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'cron'@'10.0.0.12';

-- From Dialer 3 (10.0.0.13)
CREATE USER 'cron'@'10.0.0.13' IDENTIFIED BY 'Str0ng_Cr0n_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'cron'@'10.0.0.13';

-- User: custom (for custom API/integration scripts)
CREATE USER 'custom'@'10.0.0.2' IDENTIFIED BY 'Str0ng_Cust0m_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'custom'@'10.0.0.2';

CREATE USER 'custom'@'10.0.0.11' IDENTIFIED BY 'Str0ng_Cust0m_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'custom'@'10.0.0.11';

CREATE USER 'custom'@'10.0.0.12' IDENTIFIED BY 'Str0ng_Cust0m_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'custom'@'10.0.0.12';

CREATE USER 'custom'@'10.0.0.13' IDENTIFIED BY 'Str0ng_Cust0m_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'custom'@'10.0.0.13';

-- OPTIONAL: Wildcard grant for entire subnet (less secure, but simpler)
-- Use this if you frequently add/remove servers
-- CREATE USER 'cron'@'10.0.0.%' IDENTIFIED BY 'Str0ng_Cr0n_P@ss!';
-- GRANT ALL PRIVILEGES ON asterisk.* TO 'cron'@'10.0.0.%';

-- Keep the localhost user for the DB server itself
-- (ViciDial scripts on the DB server use localhost)
GRANT ALL PRIVILEGES ON asterisk.* TO 'cron'@'localhost';

-- Apply changes
FLUSH PRIVILEGES;

vi /etc/astguiclient.conf

# /etc/astguiclient.conf - Database Server

# Paths
PATHhome => /usr/share/astguiclient
PATHlogs => /var/log/astguiclient
PATHagi => /var/lib/asterisk/agi-bin
PATHweb => /srv/www/htdocs
PATHsounds => /var/lib/asterisk/sounds
PATHmonitor => /var/spool/asterisk/monitor
PATHDONEmonitor => /var/spool/asterisk/monitorDONE

# THIS server's IP (the DB server's LAN IP)
VARserver_ip => 10.0.0.1

# Database connection -- localhost since DB is local
VARDB_server => localhost
VARDB_database => asterisk
VARDB_user => cron
VARDB_pass => Str0ng_Cr0n_P@ss!
VARDB_custom_user => custom
VARDB_custom_pass => Str0ng_Cust0m_P@ss!
VARDB_port => 3306

# Keepalives -- DB server does NOT run Asterisk
# X = no keepalive processes
VARactive_keepalives => X

# Asterisk version (even though not running Asterisk here)
VARasterisk_version => 18

# FastAGI settings (not needed on DB-only server, but required in file)
VARfastagi_log_min_servers => 3
VARfastagi_log_max_servers => 16
VARfastagi_log_min_spare_servers => 2
VARfastagi_log_max_spare_servers => 8
VARfastagi_log_max_requests => 1000
VARfastagi_log_checkfordead => 30
VARfastagi_log_checkforwait => 60

# DB Schema version -- MUST match across all servers
ExpectedDBSchema => 1736

### ==========================================================
### CRON JOBS FOR DATABASE SERVER ONLY
### ==========================================================

### Keepalive -- runs VARactive_keepalives processes
### On DB-only server with X, this is a no-op but keep it for consistency
* * * * * /usr/share/astguiclient/ADMIN_keepalive_ALL.pl

### ViciDial Hopper -- MUST run on exactly ONE server
### Populates the lead hopper for auto-dialing
* * * * * /usr/share/astguiclient/AST_VDhopper.pl -q

### Adjust GMT offsets on leads
1 1,7 * * * /usr/share/astguiclient/ADMIN_adjust_GMTnow_on_leads.pl --debug --list-settings

### Database optimization (runs during off-hours)
3 1 * * * /usr/share/astguiclient/AST_DB_optimize.pl

### Agent time log reports
2 0 * * 0 /usr/share/astguiclient/AST_agent_week.pl
22 0 * * * /usr/share/astguiclient/AST_agent_day.pl

### Archive old log tables daily (high-volume systems)
20 1 * * * /usr/share/astguiclient/ADMIN_archive_log_tables.pl --daily

### Clean up dead callbacks
25 0 * * * /usr/share/astguiclient/AST_DB_dead_cb_purge.pl --purge-non-cb --quiet

### Flush queue DB table every hour
11 * * * * /usr/share/astguiclient/AST_flush_DBqueue.pl -q

### Reset temporary info tables
2 1 * * * /usr/share/astguiclient/AST_reset_mysql_vars.pl

### Fix agent log once per hour and full day at night
33 * * * * /usr/share/astguiclient/AST_cleanup_agent_log.pl
50 0 * * * /usr/share/astguiclient/AST_cleanup_agent_log.pl --last-24hours

### Compress and rotate astguiclient logs
25 2 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +1 -print | grep -v \.xz | xargs xz -9 >/dev/null 2>&1
28 0 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +30 -print | xargs rm -f

### Email processing -- MUST run on exactly ONE server
### Uncomment if using inbound email features
#* * * * * /usr/share/astguiclient/AST_inbound_email_parser.pl

# 1. MariaDB is running and listening on network
ss -tlnp | grep 3306

# 2. Can connect locally
mysql -u cron -p'Str0ng_Cr0n_P@ss!' -e "SELECT COUNT(*) FROM asterisk.servers;"

# 3. NTP is synchronized
chronyc tracking

# 4. Cron is active
crontab -l | grep -c "astguiclient"
# Should show 10+ entries

systemctl status apache2
php -v

dnf install httpd php php-mysqlnd php-cli php-gd php-xml php-mbstring php-json php-opcache
systemctl enable httpd
systemctl start httpd

zypper install apache2 php8 php8-mysql php8-gd php8-mbstring php8-xmlwriter php8-opcache
a2enmod php8
systemctl enable apache2
systemctl start apache2

php -m | grep -iE 'mysql|gd|mbstring|xml|json|curl|openssl|session'

cd /usr/src/astguiclient/trunk

# Run the install script
# When prompted, specify:
#   - Server IP: this web server's IP (e.g., 10.0.0.2)
#   - DB server: the database server IP (e.g., 10.0.0.1)
perl install.pl

# The installer copies web files to PATHweb (default: /srv/www/htdocs/)
# and creates /etc/astguiclient.conf

vi /etc/astguiclient.conf

# /etc/astguiclient.conf - Web Server

# Paths
PATHhome => /usr/share/astguiclient
PATHlogs => /var/log/astguiclient
PATHagi => /var/lib/asterisk/agi-bin
PATHweb => /srv/www/htdocs
PATHsounds => /var/lib/asterisk/sounds
PATHmonitor => /var/spool/asterisk/monitor
PATHDONEmonitor => /var/spool/asterisk/monitorDONE

# THIS server's IP
VARserver_ip => 10.0.0.2

# CRITICAL: Point to the REMOTE database server
VARDB_server => 10.0.0.1
VARDB_database => asterisk
VARDB_user => cron
VARDB_pass => Str0ng_Cr0n_P@ss!
VARDB_custom_user => custom
VARDB_custom_pass => Str0ng_Cust0m_P@ss!
VARDB_port => 3306

# Web server does NOT run Asterisk -- no keepalive processes
VARactive_keepalives => X

VARasterisk_version => 18

VARfastagi_log_min_servers => 3
VARfastagi_log_max_servers => 16
VARfastagi_log_min_spare_servers => 2
VARfastagi_log_max_spare_servers => 8
VARfastagi_log_max_requests => 1000
VARfastagi_log_checkfordead => 30
VARfastagi_log_checkforwait => 60

ExpectedDBSchema => 1736

# ViciBox default location:
cat /srv/www/htdocs/vicidial/dbconnect_mysqli.php

# AlmaLinux default:
cat /var/www/html/vicidial/dbconnect_mysqli.php

<VirtualHost *:80>
    DocumentRoot /srv/www/htdocs
    ServerName vici-web1.local

    <Directory /srv/www/htdocs>
        Options -Indexes +FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>

    <Directory /srv/www/htdocs/vicidial>
        Options -Indexes
        AllowOverride All
        Require all granted
    </Directory>

    # PHP settings for ViciDial
    <IfModule mod_php.c>
        php_value max_execution_time 330
        php_value max_input_time 300
        php_value memory_limit 256M
        php_value post_max_size 64M
        php_value upload_max_filesize 64M
        php_value session.gc_maxlifetime 28800
        php_value error_reporting 22519
    </IfModule>

    ErrorLog /var/log/apache2/vicidial_error.log
    CustomLog /var/log/apache2/vicidial_access.log combined
</VirtualHost>

DocumentRoot /var/www/html
ErrorLog /var/log/httpd/vicidial_error.log
CustomLog /var/log/httpd/vicidial_access.log combined

systemctl restart apache2   # openSUSE
# or
systemctl restart httpd     # AlmaLinux

### ==========================================================
### CRON JOBS FOR WEB SERVER ONLY
### ==========================================================

### Keepalive -- with VARactive_keepalives => X, this is a no-op
### Keep it for consistency in case you add web-related processes later
* * * * * /usr/share/astguiclient/ADMIN_keepalive_ALL.pl

### Compress/rotate astguiclient logs
25 2 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +1 -print | grep -v \.xz | xargs xz -9 >/dev/null 2>&1
28 0 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +30 -print | xargs rm -f

# On web server
mkdir -p /var/spool/asterisk/monitorDONE

# Mount from each dialer (add to /etc/fstab)
10.0.0.11:/var/spool/asterisk/monitorDONE /mnt/recordings-dial1 nfs defaults,ro 0 0
10.0.0.12:/var/spool/asterisk/monitorDONE /mnt/recordings-dial2 nfs defaults,ro 0 0

# 1. Apache is running
systemctl status apache2   # or httpd

# 2. Can connect to remote database
mysql -h 10.0.0.1 -u cron -p'Str0ng_Cr0n_P@ss!' -e "SELECT COUNT(*) FROM asterisk.vicidial_users;"

# 3. ViciDial admin login works
curl -s http://10.0.0.2/vicidial/admin.php | grep -i "login"
# Should return the login page HTML

# 4. PHP has required modules
php -m | grep -c "mysqli\|gd\|mbstring"
# Should return 3

asterisk -V
# Asterisk 18.x.x-vici or Asterisk 20.x.x-vici

cd /usr/src
# Download ViciDial-patched Asterisk source
# (Check the ViciDial forum for the latest recommended version)
wget http://download.vicidial.com/required-apps/asterisk-18-current-vici.tar.gz
tar xzf asterisk-18-current-vici.tar.gz
cd asterisk-18.*/
./configure --with-jansson-bundled
make menuselect    # Enable: app_meetme, app_confbridge, res_timing_dahdi
make
make install
make samples
make config        # Install init scripts

cd /usr/src
wget http://download.vicidial.com/required-apps/dahdi-linux-current.tar.gz
tar xzf dahdi-linux-current.tar.gz
cd dahdi-linux-*/
make
make install

# Load timing module
modprobe dahdi
dahdi_genconf

vi /etc/astguiclient.conf

# /etc/astguiclient.conf - Dialer Server 1

# Paths (standard ViciBox paths)
PATHhome => /usr/share/astguiclient
PATHlogs => /var/log/astguiclient
PATHagi => /var/lib/asterisk/agi-bin
PATHweb => /srv/www/htdocs
PATHsounds => /var/lib/asterisk/sounds
PATHmonitor => /var/spool/asterisk/monitor
PATHDONEmonitor => /var/spool/asterisk/monitorDONE

# THIS server's IP -- MUST match the IP registered in ViciDial admin
VARserver_ip => 10.0.0.11

# CRITICAL: Point to the REMOTE database server
VARDB_server => 10.0.0.1
VARDB_database => asterisk
VARDB_user => cron
VARDB_pass => Str0ng_Cr0n_P@ss!
VARDB_custom_user => custom
VARDB_custom_pass => Str0ng_Cust0m_P@ss!
VARDB_port => 3306

# Keepalive processes to run on this DIALER server
# 1 = AST_update (Asterisk state updates)
# 2 = AST_send_listen (send/listen audio monitoring)
# 3 = AST_VDauto_dial (auto-dialer -- runs on ALL dialer servers)
# 4 = AST_VDremote_agents (remote agent handling)
# 5 = AST_VDadapt (ONLY on ONE server in the entire cluster)
# 6 = FastAGI_log (FastAGI logging daemon)
# 7 = AST_VDauto_dial_FILL (ONLY on ONE server in the cluster)
# 8 = ip_relay (blind monitoring relay)
# 9 = Timeclock auto logout (ONLY on ONE server in the cluster)
# C = ConfBridge process
# E = Email processor (ONLY on ONE server in the cluster)
# S = SIP Logger (Asterisk 13+ with patches)

# Dialer 1 -- the "primary" dialer, runs the singleton processes
VARactive_keepalives => 12345679C

# NOTE: Dialer 2 and 3 would use: 12346C
# (No 5, 7, or 9 -- those are singletons running on Dialer 1)

VARasterisk_version => 18

# FTP for recording archival (optional)
VARFTP_host => 10.0.0.1
VARFTP_user => cron
VARFTP_pass => test
VARFTP_port => 21
VARFTP_dir => RECORDINGS
VARHTTP_path => http://10.0.0.11/RECORDINGS/MP3/

# FastAGI settings
VARfastagi_log_min_servers => 3
VARfastagi_log_max_servers => 16
VARfastagi_log_min_spare_servers => 2
VARfastagi_log_max_spare_servers => 8
VARfastagi_log_max_requests => 1000
VARfastagi_log_checkfordead => 30
VARfastagi_log_checkforwait => 60

ExpectedDBSchema => 1736

DB Server:       X               (no Asterisk processes)
Web Server:      X               (no Asterisk processes)
Dialer 1:       12345679C        (primary, runs singletons)
Dialer 2:       12346C           (secondary)
Dialer 3:       12346C           (secondary)

cd /usr/src/astguiclient/trunk

# Install ViciDial scripts and AGI files
perl install.pl

# When prompted:
#   Server IP: 10.0.0.11 (this dialer's IP)
#   Database Server: 10.0.0.1 (the central DB)
#   Database Name: asterisk
#   Database User: cron
#   Database Pass: (your password)

### ==========================================================
### CRON JOBS FOR DIALER SERVER 1 (PRIMARY)
### ==========================================================

### Keepalive -- restarts crashed processes defined in VARactive_keepalives
* * * * * /usr/share/astguiclient/ADMIN_keepalive_ALL.pl

### Compress/rotate astguiclient logs
25 2 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +1 -print | grep -v \.xz | xargs xz -9 >/dev/null 2>&1
28 0 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +30 -print | xargs rm -f

### Kill hung/congested Asterisk channels
* * * * * /usr/share/astguiclient/AST_manager_kill_hung_congested.pl

### Voicemail updater
* * * * * /usr/share/astguiclient/AST_vm_update.pl

### Conference validator
* * * * * /usr/share/astguiclient/AST_conf_update.pl

### Recording mixing and compression (runs every 3 minutes)
0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57 * * * * /usr/share/astguiclient/AST_CRON_audio_1_move_mix.pl --MIX
0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57 * * * * /usr/share/astguiclient/AST_CRON_audio_1_move_VDonly.pl
1,4,7,10,13,16,19,22,25,28,31,34,37,40,43,46,49,52,55,58 * * * * /usr/share/astguiclient/AST_CRON_audio_2_compress.pl --MP3

### FTP recordings to archive server (uncomment if using FTP archival)
#2,5,8,11,14,17,20,23,26,29,32,35,38,41,44,47,50,53,56,59 * * * * /usr/share/astguiclient/AST_CRON_audio_3_ftp.pl --MP3 --run-check

### Remove old recordings (adjust retention as needed)
#24 0 * * * /usr/bin/find /var/spool/asterisk/monitorDONE -maxdepth 2 -type f -mtime +90 -print | xargs rm -f
24 1 * * * /usr/bin/find /var/spool/asterisk/monitorDONE/ORIG -maxdepth 2 -type f -mtime +1 -print | xargs rm -f

### Rotate old Asterisk logs
29 0 * * * /usr/bin/find /var/log/asterisk -maxdepth 3 -type f -mtime +30 -print | xargs rm -f
30 0 * * * /usr/bin/find / -maxdepth 1 -name "screenlog.0*" -mtime +7 -print | xargs rm -f
31 0 * * * /usr/bin/find /tmp -maxdepth 1 -type f -mtime +7 -print | xargs rm -f
32 0 * * * /usr/bin/find /var/log/asterisk -maxdepth 1 -type f -mtime +1 -print | grep -v \.xz | xargs xz >/dev/null 2>&1

### ==========================================================
### CRON JOBS FOR DIALER SERVER 2/3 (SECONDARY)
### ==========================================================

### Keepalive
* * * * * /usr/share/astguiclient/ADMIN_keepalive_ALL.pl

### Compress/rotate astguiclient logs
25 2 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +1 -print | grep -v \.xz | xargs xz -9 >/dev/null 2>&1
28 0 * * * /usr/bin/find /var/log/astguiclient -maxdepth 1 -type f -mtime +30 -print | xargs rm -f

### Kill hung/congested channels
* * * * * /usr/share/astguiclient/AST_manager_kill_hung_congested.pl

### Voicemail updater
* * * * * /usr/share/astguiclient/AST_vm_update.pl

### Conference validator
* * * * * /usr/share/astguiclient/AST_conf_update.pl

### Recording mixing and compression
0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57 * * * * /usr/share/astguiclient/AST_CRON_audio_1_move_mix.pl --MIX
0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57 * * * * /usr/share/astguiclient/AST_CRON_audio_1_move_VDonly.pl
1,4,7,10,13,16,19,22,25,28,31,34,37,40,43,46,49,52,55,58 * * * * /usr/share/astguiclient/AST_CRON_audio_2_compress.pl --MP3

### FTP recordings (uncomment if using)
#2,5,8,11,14,17,20,23,26,29,32,35,38,41,44,47,50,53,56,59 * * * * /usr/share/astguiclient/AST_CRON_audio_3_ftp.pl --MP3 --run-check

### Remove old recordings
#24 0 * * * /usr/bin/find /var/spool/asterisk/monitorDONE -maxdepth 2 -type f -mtime +90 -print | xargs rm -f
24 1 * * * /usr/bin/find /var/spool/asterisk/monitorDONE/ORIG -maxdepth 2 -type f -mtime +1 -print | xargs rm -f

### Rotate Asterisk logs
29 0 * * * /usr/bin/find /var/log/asterisk -maxdepth 3 -type f -mtime +30 -print | xargs rm -f
30 0 * * * /usr/bin/find / -maxdepth 1 -name "screenlog.0*" -mtime +7 -print | xargs rm -f
31 0 * * * /usr/bin/find /tmp -maxdepth 1 -type f -mtime +7 -print | xargs rm -f
32 0 * * * /usr/bin/find /var/log/asterisk -maxdepth 1 -type f -mtime +1 -print | grep -v \.xz | xargs xz >/dev/null 2>&1

# 1. Asterisk is running
asterisk -V
systemctl status asterisk

# 2. Can connect to remote database
mysql -h 10.0.0.1 -u cron -p'Str0ng_Cr0n_P@ss!' -e "SELECT COUNT(*) FROM asterisk.servers;"

# 3. Keepalive processes are running (after 1-2 minutes)
screen -ls
# Should show: AST_update, AST_send_listen, AST_VDauto_dial, FastAGI_log, etc.

# 4. FastAGI is listening
ss -tlnp | grep 4577
# Should show a Perl process on 127.0.0.1:4577

# 5. NTP is synchronized
chronyc tracking

[general]
bindport = 4569
bindaddr = 0.0.0.0
delayreject = yes
adsi = no
language = en
disallow = all
allow = ulaw
allow = alaw
allow = gsm
jitterbuffer = no
forcejitterbuffer = no
maxjitterbuffer = 1000
maxjitterinterps = 10
resyncthreshold = -1
trunkfreq = 20
trunktimestamps = yes
minregexpire = 60
maxregexpire = 360
iaxthreadcount = 50
iaxmaxthreadcount = 250
authdebug = no
regcontext = iaxregistration
autokill = yes
codecpriority = host
allowfwdownload = no
context = trunkinbound

#include iax-vicidial.conf

; ---- CLUSTER IAX PEERS ----

; Peer: Dialer 2
[dialer2]
type = friend
context = default
auth = plaintext
secret = ClusterS3cret!
host = 10.0.0.12
disallow = all
allow = ulaw
allow = alaw
trunk = yes
qualify = yes
requirecalltoken = no
jitterbuffer = no

; Peer: Dialer 3
[dialer3]
type = friend
context = default
auth = plaintext
secret = ClusterS3cret!
host = 10.0.0.13
disallow = all
allow = ulaw
allow = alaw
trunk = yes
qualify = yes
requirecalltoken = no
jitterbuffer = no

; (same [general] section as above)

#include iax-vicidial.conf

; ---- CLUSTER IAX PEERS ----

; Peer: Dialer 1
[dialer1]
type = friend
context = default
auth = plaintext
secret = ClusterS3cret!
host = 10.0.0.11
disallow = all
allow = ulaw
allow = alaw
trunk = yes
qualify = yes
requirecalltoken = no
jitterbuffer = no

; Peer: Dialer 3
[dialer3]
type = friend
context = default
auth = plaintext
secret = ClusterS3cret!
host = 10.0.0.13
disallow = all
allow = ulaw
allow = alaw
trunk = yes
qualify = yes
requirecalltoken = no
jitterbuffer = no

; (same [general] section as above)

#include iax-vicidial.conf

; Peer: Dialer 1
[dialer1]
type = friend
context = default
auth = plaintext
secret = ClusterS3cret!
host = 10.0.0.11
disallow = all
allow = ulaw
allow = alaw
trunk = yes
qualify = yes
requirecalltoken = no
jitterbuffer = no

; Peer: Dialer 2
[dialer2]
type = friend
context = default
auth = plaintext
secret = ClusterS3cret!
host = 10.0.0.12
disallow = all
allow = ulaw
allow = alaw
trunk = yes
qualify = yes
requirecalltoken = no
jitterbuffer = no

register => dialer1:ClusterS3cret!@10.0.0.12
register => dialer1:ClusterS3cret!@10.0.0.13

register => dialer2:ClusterS3cret!@10.0.0.11
register => dialer2:ClusterS3cret!@10.0.0.13

register => dialer3:ClusterS3cret!@10.0.0.11
register => dialer3:ClusterS3cret!@10.0.0.12

# On each dialer:
asterisk -rx "iax2 reload"

# Verify peers are registered:
asterisk -rx "iax2 show peers"
# Should show the other dialers with OK status

# Verify registrations:
asterisk -rx "iax2 show registry"
# Should show Registered status for each peer

# From Dialer 1, check peers:
asterisk -rx "iax2 show peers"

# Expected output:
# Name/Username  Host           (S)    Mask             Port    Status     Description
# dialer2        10.0.0.12       (S)    255.255.255.255  4569    OK (1 ms)
# dialer3        10.0.0.13       (S)    255.255.255.255  4569    OK (2 ms)

# Test a call between servers (from Asterisk CLI):
# asterisk -rx "channel originate IAX2/dialer2:ClusterS3cret!@10.0.0.12/s@default application Playback hello-world"

http://10.0.0.2/vicidial/admin.php

-- Insert conference entries for Dialer 1
-- Standard conferences (conferences table)
INSERT INTO conferences (conf_exten, server_ip, extension)
SELECT seq, '10.0.0.11', ''
FROM (
    SELECT 8600001 + seq.n AS seq
    FROM (
        SELECT a.N + b.N * 10 AS n
        FROM (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4
              UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) a,
             (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2 UNION SELECT 3) b
    ) seq
    WHERE 8600001 + seq.n <= 8600039
) t;

-- ViciDial conferences (vicidial_conferences table)
INSERT INTO vicidial_conferences (conf_exten, server_ip, extension, leave_3way)
SELECT seq, '10.0.0.11', '', 'ENABLED'
FROM (
    SELECT 8600051 + seq.n AS seq
    FROM (
        SELECT a.N + b.N * 10 + c.N * 100 AS n
        FROM (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4
              UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) a,
             (SELECT 0 AS N UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4
              UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) b,
             (SELECT 0 AS N UNION SELECT 1) c
        ) seq
    WHERE 8600051 + seq.n <= 8600250
) t;

-- Repeat for Dialer 2 (10.0.0.12) with ranges 8600040-8600078 and 8600251-8600450
-- Repeat for Dialer 3 (10.0.0.13) with ranges 8600079-8600117 and 8600451-8600650

/etc/asterisk/extensions.conf          -- Main file, includes others
/etc/asterisk/extensions-vicidial.conf -- Auto-generated by ViciDial
/etc/asterisk/customexte.conf          -- Custom extensions (ring groups, etc.)

[globals]
; Local loopback trunks (auto-generated in iax-vicidial.conf, included here)
; TRUNKloop, TRUNKblind, TRUNKplay are defined by ViciDial

; Cross-server IAX trunks for cluster load balancing
; Format: IAX2/peername:secret@remote_ip
TRUNKIAX2 = IAX2/dialer1:ClusterS3cret!@10.0.0.12
TRUNKIAX3 = IAX2/dialer1:ClusterS3cret!@10.0.0.13

[globals]
TRUNKIAX1 = IAX2/dialer2:ClusterS3cret!@10.0.0.11
TRUNKIAX3 = IAX2/dialer2:ClusterS3cret!@10.0.0.13

[globals]
TRUNKIAX1 = IAX2/dialer3:ClusterS3cret!@10.0.0.11
TRUNKIAX2 = IAX2/dialer3:ClusterS3cret!@10.0.0.12

; --- CROSS-SERVER CONFERENCE ROUTING ---

; Local server (10.0.0.11) -- route locally via Goto
exten => _010*000*000*011*8600XXX,1,Goto(default,${EXTEN:16},1)
exten => _010*000*000*011*8600XXX,2,Hangup()
exten => _**010*000*000*011*8600XXX,1,Goto(default,${EXTEN:18},1)
exten => _**010*000*000*011*8600XXX,2,Hangup()

; Remote: Dialer 2 (10.0.0.12) -- route via IAX trunk
exten => _010*000*000*012*8600XXX,1,Dial(${TRUNKIAX2}/${EXTEN:16},55,o)
exten => _010*000*000*012*8600XXX,2,Hangup()
exten => _**010*000*000*012*8600XXX,1,Dial(${TRUNKIAX2}/${EXTEN:18},55,o)
exten => _**010*000*000*012*8600XXX,2,Hangup()

; Remote: Dialer 3 (10.0.0.13) -- route via IAX trunk
exten => _010*000*000*013*8600XXX,1,Dial(${TRUNKIAX3}/${EXTEN:16},55,o)
exten => _010*000*000*013*8600XXX,2,Hangup()
exten => _**010*000*000*013*8600XXX,1,Dial(${TRUNKIAX3}/${EXTEN:18},55,o)
exten => _**010*000*000*013*8600XXX,2,Hangup()

; CRITICAL: Conference transfer fix AGI -- must appear on ALL servers
; This MUST come BEFORE the IP-encoded patterns in priority
exten => _8600XXX*.,1,AGI(agi-VDADfixCXFER.agi)
exten => _8600XXX*.,n,Hangup()
exten => _78600XXX*.,1,AGI(agi-VDADfixCXFER.agi)
exten => _78600XXX*.,n,Hangup()

; Local (10.0.0.12) -- route locally
exten => _010*000*000*012*8600XXX,1,Goto(default,${EXTEN:16},1)
exten => _010*000*000*012*8600XXX,2,Hangup()
exten => _**010*000*000*012*8600XXX,1,Goto(default,${EXTEN:18},1)
exten => _**010*000*000*012*8600XXX,2,Hangup()

; Remote: Dialer 1 (10.0.0.11) -- route via IAX
exten => _010*000*000*011*8600XXX,1,Dial(${TRUNKIAX1}/${EXTEN:16},55,o)
exten => _010*000*000*011*8600XXX,2,Hangup()
exten => _**010*000*000*011*8600XXX,1,Dial(${TRUNKIAX1}/${EXTEN:18},55,o)
exten => _**010*000*000*011*8600XXX,2,Hangup()

; Remote: Dialer 3 (10.0.0.13) -- route via IAX
exten => _010*000*000*013*8600XXX,1,Dial(${TRUNKIAX3}/${EXTEN:16},55,o)
exten => _010*000*000*013*8600XXX,2,Hangup()
exten => _**010*000*000*013*8600XXX,1,Dial(${TRUNKIAX3}/${EXTEN:18},55,o)
exten => _**010*000*000*013*8600XXX,2,Hangup()

; Conference transfer fix AGI
exten => _8600XXX*.,1,AGI(agi-VDADfixCXFER.agi)
exten => _8600XXX*.,n,Hangup()
exten => _78600XXX*.,1,AGI(agi-VDADfixCXFER.agi)
exten => _78600XXX*.,n,Hangup()

; Load Balanced auto-dial (extension 8368)
; Calls this AGI 3 times for retry resilience
exten => 8368,1,AGI(agi://127.0.0.1:4577/call_log)
exten => 8368,2,AGI(agi-VDAD_LB_transfer.agi,${EXTEN})
exten => 8368,3,AGI(agi-VDAD_LB_transfer.agi,${EXTEN})
exten => 8368,4,AGI(agi-VDAD_LB_transfer.agi,${EXTEN})
exten => 8368,5,Hangup()

; Overflow auto-dial (extension 8367)
exten => 8367,1,AGI(agi://127.0.0.1:4577/call_log)
exten => 8367,2,AGI(agi-VDAD_LO_transfer.agi,${EXTEN})
exten => 8367,3,AGI(agi-VDAD_LO_transfer.agi,${EXTEN})
exten => 8367,4,AGI(agi-VDAD_LO_transfer.agi,${EXTEN})
exten => 8367,5,Hangup()

; Conference bridges for agent sessions
; Range must match what is configured in the ViciDial admin conferences table
exten => _8600XXX,1,Answer()
exten => _8600XXX,n,MeetMe(${EXTEN},Fq)
exten => _8600XXX,n,Hangup()

; ConfBridge conference bridges
exten => _9600XXX,1,Answer()
exten => _9600XXX,n,Playback(sip-silence)
exten => _9600XXX,n,ConfBridge(${EXTEN},vici_agent_bridge,vici_agent_user)
exten => _9600XXX,n,Hangup()

; /etc/asterisk/confbridge.conf

[vici_agent_bridge]
type = bridge
max_members = 10

[vici_agent_user]
type = user
quiet = yes
announce_user_count = no
music_on_hold_when_empty = no

[vici_audio_user]
type = user
quiet = yes
startmuted = yes

[vici_monitor_user]
type = user
quiet = yes
startmuted = yes

[vici_barge_user]
type = user
quiet = yes

[vici_monitor_menu]
type = menu

; Playback of recorded prompts
exten => _851XXXXX,1,Answer()
exten => _851XXXXX,n,Playback(${EXTEN})
exten => _851XXXXX,n,Hangup()

; AMD message playback
exten => _6851XXXXX,1,Answer()
exten => _6851XXXXX,n,WaitForSilence(2000,1,90)
exten => _6851XXXXX,n,Playback(sip-silence)
exten => _6851XXXXX,n,Playback(${EXTEN:1})
exten => _6851XXXXX,n,Hangup()

; Blind monitoring
exten => _08600XXX,1,Dial(${TRUNKblind}/6${EXTEN:1},55,To)
exten => _08600XXX,n,Hangup()

; Barge monitoring
exten => 8159,1,ChanSpy(,bBq)
exten => 8159,n,Hangup()

; FastAGI call logging (hangup handler) -- in EVERY context that handles calls
exten => h,1,AGI(agi://127.0.0.1:4577/call_log--HVcauses--PRI-----NODEBUG-----${HANGUPCAUSE}-----${DIALSTATUS}-----${DIALEDTIME}-----${ANSWEREDTIME}-----${HANGUPCAUSE(${HANGUPCAUSE_KEYS()},tech)}))

# Reload the dialplan without restarting Asterisk
asterisk -rx "dialplan reload"

# Verify extensions are loaded
asterisk -rx "dialplan show default" | grep 8368
asterisk -rx "dialplan show default" | grep 8600

Agent A phone → registers to Dialer 1 (10.0.0.11)
Agent B phone → registers to Dialer 2 (10.0.0.12)
Agent C phone → registers to Dialer 1 (10.0.0.11)
Agent D phone → registers to Dialer 3 (10.0.0.13)

Dialer 1 (10.0.0.11): Phones 100-149 (50 phones)
Dialer 2 (10.0.0.12): Phones 150-199 (50 phones)
Dialer 3 (10.0.0.13): Phones 200-249 (50 phones)

-- Example: Create 50 phone entries for Dialer 2
INSERT INTO phones (extension, dialplan_number, voicemail_id, phone_ip,
    computer_ip, server_ip, login, pass, status, active, phone_type,
    fullname, company, picture, protocol, local_gmt, ASTmgrUSERNAME,
    ASTmgrSECRET, login_user, login_pass, login_campaign, conf_exten,
    phone_context, phone_ring_timeout, conf_override, codecs_list,
    codecs_with_template, carrier_id, phone_context_a_b)
VALUES
('150', '150', '150', '', '', '10.0.0.12', 'cc150', 'P@ss150!',
 'ACTIVE', 'Y', 'SIP', 'Agent Phone 150', '', '', 'SIP', '-5.00',
 'cron', '1234', '', '', '', '8600150', 'default', '60', '',
 '', '', '', 'default---agent');
-- Repeat for 151-199...

# This happens automatically via the keepalive process, but you can force it:
/usr/share/astguiclient/ADMIN_keepalive_ALL.pl --justSIPpeers

# Then reload SIP:
asterisk -rx "sip reload"

# Verify peers are loaded:
asterisk -rx "sip show peers" | head -20

# From each web server and dialer:
mysql -h 10.0.0.1 -u cron -p'Str0ng_Cr0n_P@ss!' -e "
    SELECT server_ip, active_asterisk_server, active_agent_server
    FROM asterisk.servers
    ORDER BY server_ip;"

# Expected: Shows all registered servers with correct active flags

# Check screen sessions (ViciDial runs processes in screen)
screen -ls

# Expected output (varies by VARactive_keepalives):
# ASTupdate    (AST_update)
# ASTsend      (AST_send_listen)
# ASTVDauto    (AST_VDauto_dial)
# ASTremote    (AST_VDremote_agents)
# ASTVDadapt   (AST_VDadapt)           -- Only on primary dialer
# ASTfastlog   (FastAGI_log)
# ASTVDautofill (AST_VDauto_dial_FILL) -- Only on primary dialer
# ASTconf3way  (ConfBridge or MeetMe watcher)

# Verify FastAGI is listening
ss -tlnp | grep 4577

# Verify Asterisk AMI is accessible
ss -tlnp | grep 5038

# Check Asterisk is running
asterisk -rx "core show version"

# Check IAX peers
asterisk -rx "iax2 show peers"
# All cluster peers should show OK status

# Check IAX registrations
asterisk -rx "iax2 show registry"
# All registrations should show "Registered"

# Test a cross-server audio path
# From Dialer 1, call a test extension on Dialer 2:
asterisk -rx "channel originate IAX2/dialer2:ClusterS3cret!@10.0.0.12/s application Echo"

# Verify conference extensions are loaded
asterisk -rx "dialplan show default" | grep "8600"

# Check conference entries in database
mysql -h 10.0.0.1 -u cron -p'Str0ng_Cr0n_P@ss!' -e "
    SELECT server_ip, COUNT(*) as conf_count
    FROM asterisk.vicidial_conferences
    GROUP BY server_ip;"

# Expected: Each dialer IP with its conference count (200 each)

# Set max_vicidial_trunks to 0 on Dialer 1 (disables its auto-dial)
mysql -h 10.0.0.1 -u cron -p'Str0ng_Cr0n_P@ss!' -e "
    UPDATE asterisk.servers
    SET max_vicidial_trunks = 0
    WHERE server_ip = '10.0.0.11';"

# Start a test campaign and verify calls originate from Dialer 2 and 3 only

# Re-enable Dialer 1
mysql -h 10.0.0.1 -u cron -p'Str0ng_Cr0n_P@ss!' -e "
    UPDATE asterisk.servers
    SET max_vicidial_trunks = 60
    WHERE server_ip = '10.0.0.11';"

[ ] Database accessible from all servers
[ ] All servers appear in ViciDial Admin > Servers
[ ] Conference entries exist for each server (conferences + vicidial_conferences)
[ ] IAX peers show OK status on all dialers
[ ] FastAGI (port 4577) listening on all dialers
[ ] Screen processes running on all dialers
[ ] Agent can log in on each dialer server's phones
[ ] Cross-server transfers work
[ ] Recordings are accessible from web interface
[ ] NTP synchronized (< 1 second drift between all nodes)
[ ] Cron jobs installed on correct servers
[ ] VARactive_keepalives correct per server role
[ ] VDAD extension 8368 set in campaigns and server settings
[ ] Firewall rules allow inter-server communication

CREATE USER 'cron'@'10.0.0.14' IDENTIFIED BY 'Str0ng_Cr0n_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'cron'@'10.0.0.14';
CREATE USER 'custom'@'10.0.0.14' IDENTIFIED BY 'Str0ng_Cust0m_P@ss!';
GRANT ALL PRIVILEGES ON asterisk.* TO 'custom'@'10.0.0.14';
FLUSH PRIVILEGES;

-- Standard conferences
INSERT INTO conferences (conf_exten, server_ip, extension)
VALUES (8600118, '10.0.0.14', ''),
       (8600119, '10.0.0.14', ''),
       -- ... through 8600156
       ;

-- ViciDial conferences
INSERT INTO vicidial_conferences (conf_exten, server_ip, extension, leave_3way)
VALUES (8600651, '10.0.0.14', '', 'ENABLED'),
       (8600652, '10.0.0.14', '', 'ENABLED'),
       -- ... through 8600850
       ;

[dialer4]
type = friend
context = default
auth = plaintext
secret = ClusterS3cret!
host = 10.0.0.14
disallow = all
allow = ulaw
allow = alaw
trunk = yes
qualify = yes
requirecalltoken = no
jitterbuffer = no

Dialer 1: Conferences 8600001-8600039,  ViciDial 8600051-8600250   (200 each)
Dialer 2: Conferences 8600040-8600078,  ViciDial 8600251-8600450   (200 each)
Dialer 3: Conferences 8600079-8600117,  ViciDial 8600451-8600650   (200 each)
Dialer 4: Conferences 8600118-8600156,  ViciDial 8600651-8600850   (200 each)
Dialer 5: Conferences 8600157-8600195,  ViciDial 8600851-8601050   (200 each)

conferences_needed = (max_agents_on_server * 2) + max_auto_dial_trunks + 50

# Check MariaDB is running and accepting connections
mysqladmin -u cron -p'Str0ng_Cr0n_P@ss!' ping
# Should return: mysqld is alive

# Check connection count
mysql -u cron -p'Str0ng_Cr0n_P@ss!' -e "SHOW STATUS LIKE 'Threads_connected';"

# Check slow queries
mysql -u cron -p'Str0ng_Cr0n_P@ss!' -e "SHOW STATUS LIKE 'Slow_queries';"

# Check Asterisk is running
asterisk -rx "core show version"

# Check active channels
asterisk -rx "core show channels count"

# Check IAX trunk status
asterisk -rx "iax2 show peers" | grep -v "OK" | grep -v "^Name"
# Should return nothing (all peers OK)

# Check screen processes
screen -ls | grep -c "Detached"
# Should match the number of keepalive processes configured

# Check FastAGI
ss -tlnp | grep 4577

# Check Apache is running
systemctl is-active apache2 || systemctl is-active httpd

# Check ViciDial admin page responds
curl -s -o /dev/null -w "%{http_code}" http://localhost/vicidial/admin.php
# Should return 200

#!/bin/bash
# ViciDial Cluster Health Check
# Run every 5 minutes via cron

ROLE="${1:-dialer}"  # dialer, db, or web
DB_HOST="10.0.0.1"
DB_USER="cron"
DB_PASS="Str0ng_Cr0n_P@ss!"
ALERT_EMAIL="admin@example.com"

ERRORS=""

# Check 1: Database connectivity
if ! mysql -h "$DB_HOST" -u "$DB_USER" -p"$DB_PASS" -e "SELECT 1" &>/dev/null; then
    ERRORS+="CRITICAL: Cannot connect to database at $DB_HOST\n"
fi

# Check 2: NTP sync
OFFSET=$(chronyc tracking 2>/dev/null | grep "Last offset" | awk '{print $4}' | tr -d '+')
if [ -n "$OFFSET" ]; then
    # Alert if offset > 2 seconds
    ABS_OFFSET=$(echo "$OFFSET" | tr -d '-')
    if (( $(echo "$ABS_OFFSET > 2" | bc -l 2>/dev/null || echo "0") )); then
        ERRORS+="WARNING: NTP offset is ${OFFSET}s\n"
    fi
fi

if [ "$ROLE" = "dialer" ]; then
    # Check 3: Asterisk running
    if ! asterisk -rx "core show version" &>/dev/null; then
        ERRORS+="CRITICAL: Asterisk is not running\n"
    fi

    # Check 4: FastAGI listening
    if ! ss -tlnp | grep -q ":4577"; then
        ERRORS+="CRITICAL: FastAGI (port 4577) is not listening\n"
    fi

    # Check 5: IAX peers
    DOWN_PEERS=$(asterisk -rx "iax2 show peers" 2>/dev/null | grep "UNREACHABLE\|UNKNOWN" | wc -l)
    if [ "$DOWN_PEERS" -gt 0 ]; then
        ERRORS+="WARNING: $DOWN_PEERS IAX peers are down\n"
    fi

    # Check 6: Screen processes
    EXPECTED_SCREENS=6  # Adjust based on VARactive_keepalives
    ACTUAL_SCREENS=$(screen -ls 2>/dev/null | grep -c "Detached")
    if [ "$ACTUAL_SCREENS" -lt "$EXPECTED_SCREENS" ]; then
        ERRORS+="WARNING: Only $ACTUAL_SCREENS of $EXPECTED_SCREENS screen sessions running\n"
    fi
fi

if [ "$ROLE" = "web" ]; then
    # Check 7: Apache running
    if ! systemctl is-active apache2 &>/dev/null && ! systemctl is-active httpd &>/dev/null; then
        ERRORS+="CRITICAL: Apache is not running\n"
    fi
fi

# Report errors
if [ -n "$ERRORS" ]; then
    HOSTNAME=$(hostname)
    echo -e "Health check failures on $HOSTNAME:\n$ERRORS" | \
        mail -s "ViciDial Cluster Alert: $HOSTNAME" "$ALERT_EMAIL" 2>/dev/null
    echo -e "ERRORS:\n$ERRORS"
    exit 1
fi

echo "OK: All checks passed"
exit 0

chmod +x /usr/local/bin/cluster-health.sh

# Add to cron on each server (adjust role parameter):
# */5 * * * * /usr/local/bin/cluster-health.sh dialer
# */5 * * * * /usr/local/bin/cluster-health.sh db
# */5 * * * * /usr/local/bin/cluster-health.sh web

# /etc/promtail/promtail.yml (on each server)
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://10.0.0.2:3100/loki/api/v1/push

scrape_configs:
  - job_name: asterisk
    static_configs:
      - targets: [localhost]
        labels:
          job: asterisk
          host: vici-dial1
          __path__: /var/log/asterisk/messages

  - job_name: astguiclient
    static_configs:
      - targets: [localhost]
        labels:
          job: astguiclient
          host: vici-dial1
          __path__: /var/log/astguiclient/*.log

# On each ViciDial server, add to /etc/rsyslog.conf:
*.* @10.0.0.2:514

# Daily full backup with mysqldump
0 2 * * * /usr/bin/mysqldump -u root --single-transaction --routines --triggers \
    asterisk | gzip > /backup/vicidial-db-$(date +\%Y\%m\%d).sql.gz

# Keep 14 days of backups
0 3 * * * find /backup -name "vicidial-db-*.sql.gz" -mtime +14 -delete

# Backup configuration files
0 2 * * * tar czf /backup/config-$(hostname)-$(date +\%Y\%m\%d).tar.gz \
    /etc/asterisk/ /etc/astguiclient.conf /etc/crontab \
    /var/spool/cron/tabs/root 2>/dev/null

# Test connectivity from the failing server
mysql -h 10.0.0.1 -u cron -p'Str0ng_Cr0n_P@ss!' -e "SELECT 1;"

# If it fails, check:
# 1. Firewall on DB server
iptables -L -n | grep 3306

# 2. MariaDB bind address
grep bind-address /etc/my.cnf

# 3. User grants
mysql -u root -p -e "SELECT user, host FROM mysql.user WHERE user='cron';"

# 4. Network connectivity
ping -c 3 10.0.0.1
telnet 10.0.0.1 3306

# On each server, run simultaneously and compare:
date +%s.%N

# Check chrony status
chronyc tracking
chronyc sources

# Check if chrony is running
systemctl status chronyd

# Force immediate sync
chronyc makestep

# If drift persists, restart chrony
systemctl restart chronyd

# On the affected dialer
asterisk -rx "iax2 show peers"
asterisk -rx "iax2 show registry"

# Check UDP 4569 connectivity
nmap -sU -p 4569 10.0.0.12

# Check firewall
iptables -L -n | grep 4569

-- Check for overlapping conference ranges
SELECT conf_exten, GROUP_CONCAT(server_ip) as servers, COUNT(*) as cnt
FROM vicidial_conferences
GROUP BY conf_exten
HAVING cnt > 1;
-- Should return ZERO rows

-- Check conference assignments per server
SELECT server_ip, MIN(conf_exten), MAX(conf_exten), COUNT(*)
FROM vicidial_conferences
GROUP BY server_ip;

# On the dialer, verify recordings exist
ls -la /var/spool/asterisk/monitorDONE/MP3/ | tail -5

# On the web server, check if recordings are accessible
ls -la /mnt/recordings-dial1/ 2>/dev/null || echo "NFS mount missing"

# Check FTP transfer cron
grep "audio_3_ftp" /var/spool/cron/tabs/root

# Is the hopper cron running? (Should be on DB server only)
ps aux | grep VDhopper

# Check hopper directly
mysql -h 10.0.0.1 -u cron -p'Str0ng_Cr0n_P@ss!' -e "
    SELECT campaign_id, COUNT(*) FROM asterisk.vicidial_hopper
    GROUP BY campaign_id;"

# Check for hopper errors
tail -50 /var/log/astguiclient/hopper.log

   ls -la /var/lib/asterisk/agi-bin/agi-VDADfixCXFER.agi

SELECT user, server_ip, status, campaign_id
FROM vicidial_live_agents
WHERE user = 'agent123';

UPDATE phones SET server_ip = '10.0.0.12' WHERE login = 'cc150';

# Run on EVERY dialer:
grep "VARactive_keepalives" /etc/astguiclient.conf

# Kill the duplicate process
screen -ls | grep "VDadapt\|VDautofill"
# Kill the screen session if found on a secondary server

# Remove from keepalives in /etc/astguiclient.conf
# Change: VARactive_keepalives => 12345679C
# To:     VARactive_keepalives => 12346C

# === DATABASE ===
# Check connections from cluster nodes
mysql -e "SELECT substring_index(host,':',1) as client_ip, COUNT(*) as connections
    FROM information_schema.processlist
    GROUP BY client_ip;"

# Check replication lag (if using replicas)
mysql -e "SHOW SLAVE STATUS\G" | grep -E "Seconds_Behind|IO_Running|SQL_Running"

# === ASTERISK ===
# Show active channels
asterisk -rx "core show channels"

# Show SIP peers with status
asterisk -rx "sip show peers" | grep -v "OK" | grep -v "^Name"

# Show IAX peers
asterisk -rx "iax2 show peers"

# Show active conferences (MeetMe)
asterisk -rx "meetme list"

# Show active conferences (ConfBridge)
asterisk -rx "confbridge list"

# Check dialplan for specific extension
asterisk -rx "dialplan show 8368@default"

# === VICIDIAL PROCESSES ===
# Show all screen sessions
screen -ls

# Attach to a specific screen (read-only, safe)
screen -x ASTVDauto

# Check keepalive log
tail -50 /var/log/astguiclient/keepalive.log

# Check auto-dial log
tail -50 /var/log/astguiclient/VDauto_dial.log

# === NETWORK ===
# Check for established DB connections
ss -tn | grep ":3306"

# Check IAX connections
ss -un | grep ":4569"

# Test latency to DB
ping -c 10 10.0.0.1 | tail -1

-- Disable the failed server in the database
-- Run from the DB server or any working server with DB access
UPDATE servers SET
    active_asterisk_server = 'N',
    active_agent_server = 'N',
    max_vicidial_trunks = 0
WHERE server_ip = '10.0.0.12';

-- Move agents off the failed server (they will need to re-login)
UPDATE vicidial_live_agents SET
    status = 'PAUSED'
WHERE server_ip = '10.0.0.12';

Concurrent Agents    Dialer Servers    DB Server RAM    DB Server CPU
     1-50                 1               8-16 GB          4 cores
    50-100                2               16-32 GB         8 cores
   100-200                3               32-64 GB         8-16 cores
   200-400                4-6             64-128 GB        16-32 cores
   400+                   6+              128 GB+          32+ cores

Rules of thumb:
- 1 dialer server per 50-75 concurrent agents
- InnoDB buffer pool = 60-70% of DB server RAM
- 200 vicidial_conferences per dialer server
- SIP phones per dialer: max 100 (recommended 50-75)
- IAX trunk capacity: ~1000 concurrent calls per trunk
- DB queries per agent: ~20-40 per second during active dialing
- Total DB connections: ~5-10 per dialer + ~5 per web server + cron