PBXNSIP Service down

[Ganesh]

Hi,

 

We are running PBXNSIP version 3.0 on Linux centOS. The system was working fine and no changes were made. Its suddenly down and we have restarted it number of times but the service is not starting.

 

[root@cel-sip ~]# service httpd status

 

httpd (pid 2390 2388 2387 2386 2385 2384 2383 2382 2294) is running..

 

[root@cel-sip ~]# cd /etc/init.d

 

[root@cel-sip init.d]# ./pbxnsip restart

 

Stopping PBX:FAILED]

 

Starting PBX:

 

[root@cel-sip init.d]# ./pbxnsip status

 

pbxctrl is stopped

 

[root@cel-sip init.d]#

 

[root@cel-sip ~]# more /etc/init.d/pbxnsip

 

#!/bin/bash

 

#

 

# Init file for pbxnsip PBX

 

#

 

# Copyright © 2006 pbxnsip Inc., USA

 

#

 

# chkconfig: 2345 20 80

 

# description: SIP-based PBX

 

#

 

# processname: pbxctrl

 

# pidfile: /var/run/pbxctrl.pid

 

# source function library

 

. /etc/rc.d/init.d/functions

 

RETVAL=0

 

# Installation location

 

INSTALLDIR=/srv/pbx

 

PBX=$INSTALLDIR/pbxctrl

 

start()

 

{

 

echo -n "Starting PBX:"

 

$PBX --dir $INSTALLDIR

 

echo

 

RETVAL=1

 

}

 

stop()

 

{

 

echo -n "Stopping PBX:"

 

killproc $PBX -TERM

 

echo

 

RETVAL=1

 

}

 

case "$1" in

 

start)

 

start

 

;;

 

stop)

 

stop

 

;;

 

restart)

 

stop

 

start

 

;;

 

status)

 

status $PBX

 

RETVAL=$?

 

;;

 

*)

 

echo $"Usage: $0 {start|stop|restart|status}"

 

RETVAL=1

 

esac

 

exit $RETVAL

 

[root@cel-sip ~]#

 

Regards

Ganesh

[Vodia PBX]

We are running PBXNSIP version 3.0 on Linux centOS. The system was working fine and no changes were made. Its suddenly down and we have restarted it number of times but the service is not starting.

 

Can you start the service manually? Is this a problem with the startup script or the PBX executable?

[Tim]

[root@cel-sip ~]# service httpd status

httpd (pid 2390 2388 2387 2386 2385 2384 2383 2382 2294) is running..

 

[root@cel-sip ~]# cd /etc/init.d

[root@cel-sip init.d]# ./pbxnsip restart

Stopping PBX:FAILED]

Starting PBX:

[root@cel-sip init.d]# ./pbxnsip status

pbxctrl is stopped

 

I was going to ask if Apache HTTPD and the pbxctrl are bound to different ports, but it seems that, at least in the build that I am running, it doesn't the prevent pbxctrl process from starting. It does throw a "FATAL" error, so I would have expected it to exit, not finish starting. Do you have logging to a file enabled and does it show any errors? I am seeing the errors below when I start pbxctrl after running the following commands. We are currently running build pbxctrl-centos5-3.0.1.3014.

 

# nc -l 80 &
# nc -l 443 &
# /etc/init.d/pbxctrl start

From my pbx.log:
[0] 20080915060138: Could not bind socket to port 80 on IP 0.0.0.0
[0] 20080915060138: FATAL: Could not open TCP port 80 for HTTP/HTTPS
[0] 20080915060138: Could not bind socket to port 80 on IP [::]
[0] 20080915060138: FATAL: Could not open TCP port 80 for HTTP/HTTPS
[0] 20080915060138: Could not bind socket to port 443 on IP 0.0.0.0
[0] 20080915060138: FATAL: Could not open TCP port 443 for HTTP/HTTPS

 

Tim

[Vodia PBX]

From my pbx.log:

[0] 20080915060138: Could not bind socket to port 80 on IP 0.0.0.0

[0] 20080915060138: FATAL: Could not open TCP port 80 for HTTP/HTTPS

[0] 20080915060138: Could not bind socket to port 80 on IP [::]

[0] 20080915060138: FATAL: Could not open TCP port 80 for HTTP/HTTPS

[0] 20080915060138: Could not bind socket to port 443 on IP 0.0.0.0

[0] 20080915060138: FATAL: Could not open TCP port 443 for HTTP/HTTPS[/code]

 

Well, those errors are probably because the PBX does not have superturtle powers to open those protected ports.

[Tim]

Well, those errors are probably because the PBX does not have superturtle powers to open those protected ports.

 

I realize that, seeing as I intentionally CAUSED the errors to happen.

 

What I am surprised about is the pbxctrl process throws an error that it labels as "FATAL" and starts anyway. I would have thought that if it couldn't bind to a required port, the process would not be allowed to finish starting giving the false appearance that it is running correctly. This is kind of important since I can cause errors on the SIP ports as well and any process tests will show that pbxctrl is up and running even though something else is bound to the ports it requires to operate. Please look at the example below, pbxctrl is up and running, but not bound to any of the SIP or HTTP/HTTPS ports.

 

[root@pbx-test ~]# netstat -l -p |grep nc
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address			   Foreign Address			 State
tcp		0	  0 *:sip					   *:*						 LISTEN	  22237/nc
tcp		0	  0 *:sip-tls				   *:*						 LISTEN	  22238/nc
tcp		0	  0 *:http					  *:*						 LISTEN	  22239/nc
tcp		0	  0 *:https					 *:*						 LISTEN	  22241/nc
udp		0	  0 *:sip					   *:*									 22242/nc

[root@pbx-test ~]# /etc/init.d/pbxctrl start
Starting PBX:

# ps -C pbxctrl
 PID TTY		  TIME CMD
22249 pts/0	00:00:00 pbxctrl

# cat /opt/log/pbx.log
[7] 20080915070155: UDP: Opening socket
[7] 20080915070155: UDPv6: Opening socket
[0] 20080915070155: UDP: bind() to port 5060 failed
[0] 20080915070155: FATAL: Could not open UDP port 5060 for SIP
[7] 20080915070155: UDPv6: Opening socket on port 5060
[0] 20080915070155: Could not bind socket to port 5060 on IP 0.0.0.0
[0] 20080915070155: FATAL: Could not open TCP port 5060 for SIP
[7] 20080915070155: Opening TCPv6 socket on port 5060
[0] 20080915070155: Could not bind socket to port 5061 on IP 0.0.0.0
[0] 20080915070155: FATAL: Could not open TCP port 5061 for SIP
[7] 20080915070155: Opening TCPv6 socket on port 5061
[2] 20080915070155: Set processor affinity to 1 failed
[0] 20080915070155: Could not bind socket to port 80 on IP 0.0.0.0
[0] 20080915070155: FATAL: Could not open TCP port 80 for HTTP/HTTPS
[7] 20080915070155: Opening TCPv6 socket on port 80
[0] 20080915070155: Could not bind socket to port 443 on IP 0.0.0.0
[0] 20080915070155: FATAL: Could not open TCP port 443 for HTTP/HTTPS

 

Compare this to httpd when you start it and its ports are already in use. It throws a bunch of FATAL errors and then neatly exits.

 

[root@pbx-test ~]# /etc/init.d/httpd start
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
													   [FAILED]

 

The problem, as I see it, is the pbxctrl process will allow itself to start in a potentially inconsistent manner.

 

Tim

[Vodia PBX]

The problem, as I see it, is the pbxctrl process will allow itself to start in a potentially inconsistent manner.

 

Well, that is a policy question... If the PBX does not start because it does nto get the TFTP port, I would call that picky. HTTP is also not essential for making phone calls... So where is the line?

 

IMHO it is reasolable to start the process anyway. If you have HTTP, then you can fix the other problems through the web interface. If you have SIP, you can run the service already.

[Tim]

Well, that is a policy question... If the PBX does not start because it does nto get the TFTP port, I would call that picky. HTTP is also not essential for making phone calls... So where is the line?

 

IMHO it is reasolable to start the process anyway. If you have HTTP, then you can fix the other problems through the web interface. If you have SIP, you can run the service already.

 

That is the problem. If the HTTP port is not able to bind to the port, you can't get into the web interface to fix anything. You are also going to have to restart the PBX to get into the web interface in the future. In my test above, the SIP ports were not able to be opened, so that is a call affecting problem.

 

I would agree with you that non-critical services like TFTP and SNMP are probably not a huge issue, if perhaps a big warning could be put on the Status page. As for HTTP/HTTPS, I'm not convinced that your management interface isn't critical, but the pbxctrl process definitely not start without the SIP ports being able to bind to their ports.

 

If there was some way to restart the management interface that would not be call impacting, then I would agree with you 100% that it is not a critical process. However, since getting the HTTP interface back would require a restart of the pbxctrl process a client, ITSP or manager could get backed into a really difficult position. (i.e. Take down 50 active calls to bring the web interface back or wait hours to make this change that a client needs now to restart when the load is much lower.)

 

If you aren't comfortable with having the process not start unless all the ports bind successfully, perhaps an option can be added to the Global Configuration for Strict or Loose Startup mode. Strict being, if any port bindings fail, pbxctrl will exit with an error code. Loose being, as long as the SIP ports bind, the pbxctrl process will start properly.

 

Tim

[Vodia PBX]

That is the problem. If the HTTP port is not able to bind to the port, you can't get into the web interface to fix anything. You are also going to have to restart the PBX to get into the web interface in the future. In my test above, the SIP ports were not able to be opened, so that is a call affecting problem.

 

I would agree with you that non-critical services like TFTP and SNMP are probably not a huge issue, if perhaps a big warning could be put on the Status page. As for HTTP/HTTPS, I'm not convinced that your management interface isn't critical, but the pbxctrl process definitely not start without the SIP ports being able to bind to their ports.

 

If there was some way to restart the management interface that would not be call impacting, then I would agree with you 100% that it is not a critical process. However, since getting the HTTP interface back would require a restart of the pbxctrl process a client, ITSP or manager could get backed into a really difficult position. (i.e. Take down 50 active calls to bring the web interface back or wait hours to make this change that a client needs now to restart when the load is much lower.)

 

If you aren't comfortable with having the process not start unless all the ports bind successfully, perhaps an option can be added to the Global Configuration for Strict or Loose Startup mode. Strict being, if any port bindings fail, pbxctrl will exit with an error code. Loose being, as long as the SIP ports bind, the pbxctrl process will start properly.

 

Next version will have a command line option "--no-check-ports" that will allow the PBX to start up even if FATAL errors are reported. TFTP and SNMP will not be fatal any more. The default is that the PBX will not start up any more if the SIP or HTTP ports cannot be used by the PBX.