ndemou

BTW, checked today and it works. Thanks again

Thanks!

I'm going to a random domain, verify that its time zone is default, then go to a random service flag, verify the timezone is default again and at the top I see this: Current time for this account: 9:19:05 I then verify that the operating system is displaying the correct time (8:19:34) and timezone (EET)

Hi, when selecting "Greece, Athens" timezone the time is off by +1Hr, i.e. it displays 10:45 (GMT+3) when the actual time is 9:45(GMT+2). It doesn't matter how you set the timezone (at the system level, domain level or account level). Also the OS displays time correctly (9:45 EET). My workaround was to use Central "African time (GMT+2)".

Apparently the web password is now as important as the SIP password. Right? I mean that a hacker with a domain a user name and a web password can make any external calls the user is allowed to make.

I need a switch in front of my 2 PBX's mainly for 2 reasons: Route incoming calls to one or the other PBX and outgoing calls to the best trunk (Least Cost Routing) Do some some clever mangling of SIP headers to conform with the expectations of my trunks I'm looking at free soft-switches with commercial support. Is anybody having experience with this? Any thoughts or suggestions? Freeswitch looks suitable but I'm not sure.

Any ideas at all?

First of all many thanks for the help Now, regarding the correlation between the output of lsof and the contents of /proc/`pidof pbxctrl`/fd: No they don't match and the difference is clearly on the IP sockets (see [3]).The reason for the difference is that lsof gives 7 lines of output for each socket. So if I count only unique socket ids then both outputs match: $ lsof -nP | grep 'pbxctrl' | grep 'IPv' | awk -F' ' '{print $7 }' | sort -n | uniq | wc -l 116 $ ls -l /proc/`pidof pbxctrl`/fd | grep 'socket' | wc -l 116 A quick check[1] on my saved lsof outputs during the past few weeks reveals that this factor (x7) is constant. And another check [2] reveals that: The unique socket ids at the output of lsof (that match the number of sockets in /proc/.../fd) are increasing with the pattern mentioned in my previous post (i.e. ~4% increase during working days and no increase during weekends and holidays). By the way: when you exclude sockets the number of "files" left is more-or-less constant during these weeks that I'm testing (97 open "files" with a rare peak to at most 137 now and then). Here's a plot of unique socket ids per hour for two weeks. Weekends are colored green (it's of-course the same as above but scaled down by a factor of 7 but I like pictures :-) ______________________________ [1] ls lsof.2019-0*|sort | while read FILE; do cat $FILE | grep 'IPv' | awk -F' ' '{print $7 }' | grep -v '0t0' | sort -n | uniq -c | awk -F' ' '{print $1 }' | uniq -c | sort -nr | head -1; done [2] ls lsof.2019-0*|sort | while read FILE; do echo -n "$FILE "; cat $FILE | grep 'IPv' | awk -F' ' '{print $7 }' | grep -v '0t0' | sort -n | uniq | wc -l; done [3] $ ls -l /proc/`pidof pbxctrl`/fd | grep 'socket'|wc -l 125 $ lsof -nP | grep 'pbxctrl' | grep 'IPv'|wc -l 992

I'm logging the output of lsof for the pbxctrl process every hour for about a month. It is clear that the number of lines of output grows by about 20% per week. I've just restarted pbxctrl as a precaution. I'm attaching a graph. The y-axis is the number of lines of the output of the command: lsof | grep pbxctrl . Weekends and holidays are painted in green color. Most of my customers are businesses so I have many calls with a peak at about midday during working days and very few calls during weekends and holidays. I confess I haven't dived into the details of the output of lsof. However I'm almost sure that no matter the exact meaning it doesn't seem right for the number of these lines to grow by ~4% every working day. Most likely a bug in pbxctrl and less likely one in lsof. I need your help for further debugging though. I have kept all the log files and the full output of lsof for this period. Anything else I can do?

Hmmm good thinking. I'll have to read a bit more on lsof and linux internals. Will let you know

They certainly belong to the same process: pbxctrl with PID 1486. But that's to be expected since I'm looking specifically for pbxctrl. Not sure what you're asking here so I'll provide anything I can think of: "man lsof" reads: "TID is the task (thread) IDentification number". In the example output above the 3rd column is the TID column (empty for the first line and 1530 to 1536 for the next 7 lines). B.T.W. I see the same pattern for almost all other ports: each port appears 8 times, once for the main process and another 7 times for threads. I've restarted pbxctrl and now I'm running lsof -nP | grep pbxctrl every hour and logging the full output on a series of timestamped files. Just in case it helps.

We have a few hundred extensions in small companies. Nothing unusual I would guess (some inter-domain, plenty external calls and a conference call from time to time). Surely no multicast paging. Regarding attacks, I've run for a few minutes ngrep (a packet sniffer) looking for SIP replies that are not "SIP 200 OK" and I see nothing unusual. Regarding IPv6 ports in lsof's output they account for1352 out of 3300 lines. But I should note that they contain only 169 unique port numbers. E.g.: lsof -n| grep ^pbxctrl | grep -i ipv6 |grep 49460 pbxctrl 1486 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1530 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1531 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1532 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1533 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1534 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1535 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1536 root 456u IPv6 18445566 0t0 UDP *:49460 Should I send you log files or anything else?

Unfortunately within a week 'lsof -n | grep pbxctrl' did give more than 4000 lines of output. Usage of the system is stable so I don't like that this number keeps climbing up week by week and I'm afraid of the system crashing again. Looking at today's graphs I see these PEAKS: ~770 Registrations+Subscriptions, 52 call legs 112 http(s) 112. ~ 10 for TFTP & LDAP together In case it helps: Examining the output of ngrep on port 5060 I see that I'm sending packets to 310 IP:port couples. The output of lsof | grep pbxctrl is along the lines of this: pbxctrl 1486 1536 root 442u IPv4 18445564 0t0 UDP *:49460 pbxctrl 1486 1536 root 443u IPv4 18437010 0t0 TCP 1.2.3.4:https->9.10.11.12:53054 (ESTABLISHED) pbxctrl 1486 1536 root 444u IPv4 18445565 0t0 UDP *:49461 pbxctrl 1486 1536 root 452u IPv6 18472707 0t0 UDP *:49869 pbxctrl 1486 1536 root 453u IPv4 18506734 0t0 UDP *:62612 pbxctrl 1486 1536 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1536 root 457u IPv6 18445567 0t0 UDP *:49461 pbxctrl 1486 1536 root 458u IPv4 18472815 0t0 TCP 1.2.3.4:https->5.6.7.8:34917 (ESTABLISHED) pbxctrl 1486 1536 root 461u IPv4 18506735 0t0 UDP *:62613 pbxctrl 1486 1536 root 472u IPv4 18505204 0t0 UDP *:52692 pbxctrl 1486 1536 root 473u IPv4 18505205 0t0 UDP *:52693 pbxctrl 1486 1536 root 478u IPv4 18505208 0t0 UDP *:52192 pbxctrl 1486 1536 root 479u IPv4 18505209 0t0 UDP *:52193 pbxctrl 1486 1536 root 482u IPv4 18505038 0t0 UDP *:58870 pbxctrl 1486 1536 root 483u IPv4 18505039 0t0 UDP *:58871 pbxctrl 1486 1536 root 484u IPv4 18505042 0t0 UDP *:61862 pbxctrl 1486 1536 root 485u IPv4 18505043 0t0 UDP *:61863 What's your opinion? Would you like more data?

I've upgraded to 63.0.1 about 12 days ago and the output of lsof -n | grep pbxctrl has grown from about ~2000 lines to about ~3300 lines (IPv4 and IPv6 connections). I've set a new warning level at 4000 lines and will let you know if/when I'll hit it

Thanks that was it! I had to temporarily firewall one IP with a lot of TCP connections in order to let me connect to the webUI.

Not quite fixed after all. It works for a little while then it renders pages with huge logos and unusual layout, then after trying ctrl-F5 (firefox's shortcut to *fully* reload) it stops responding. I'm seeing these lines at the logs: [6] 20190525003418: Last message repeated 3 times [3] 20190525003418: Current number of requests 50 has reached maximum 50, connections not accepted [6] 20190525003418: 140 more requests pending to acme-v02.api.letsencrypt.org:443 [6] 20190525003419: Last message repeated 2 times [3] 20190525003419: Current number of requests 50 has reached maximum 50, connections not accepted [6] 20190525003420: 140 more requests pending to acme-v02.api.letsencrypt.org:443 [6] 20190525003422: Last message repeated 5 times [3] 20190525003422: Current number of requests 50 has reached maximum 50, connections not accepted [6] 20190525003422: 140 more requests pending to acme-v02.api.letsencrypt.org:443

FIXED! removed webUI customization (pbxwebai and webpages directory) and it works.

OS is linux/CentOS. There is not firewall (wget ...localhost...). The ports _are_ open (notice that wget reports that it gets connected but I also verified with ss). rawlogin has exactly the same behavior (I get connected but get zero bytes from the PBX). I rebooted (it's deep in the night here) with no success.

Had no problem in test server but after upgrading our production system to 63.0 we lost access to the web interface! Tried from various IPs, from same LAN and from localhost with no success. Ports 8080 and 443 are open and LISTENing, we get connected but then the pbxctrl gives zero bytes back. I'm providing the ouput of tests from localhost with wget and openssl below. Please help as soon as possible. wget -vvvv http://localhost:8080/ --2019-05-24 13:40:19-- http://localhost:8080/ Resolving localhost (localhost)... ::1, ::1, 127.0.0.1 Connecting to localhost (localhost)|::1|:8080... connected. HTTP request sent, awaiting response... No data received. Retrying. [...same error again and again...] wget -vvvv https://localhost --2019-05-24 13:28:21-- https://localhost/ Resolving localhost (localhost)... ::1, ::1, 127.0.0.1 Connecting to localhost (localhost)|::1|:443... connected. Unable to establish SSL connection. openssl s_client -debug -host localhost -port 443 CONNECTED(00000003) write to 0x21d3500 [0x21d3580] (289 bytes => 289 (0x121)) 0000 - 16 03 01 01 1c 01 00 01-18 03 03 58 2a 59 35 05 ...........X*Y5. 0010 - 31 9e c9 3a 58 b9 82 80-ad 03 9c ee cf 4b 2a 1a 1..:X........K*. 0020 - ed 50 8c 11 cf 2b 4a 98-bd be 24 00 00 ac c0 30 .P...+J...$....0 0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1 .,.(.$.......... 0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37 ...k.j.i.h.9.8.7 0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a .6.........2...* 0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../ 0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0 .+.'.#.......... 0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31 ...g.@.?.>.3.2.1 0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43 .0.........E.D.C 00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c .B.1.-.).%...... 00b0 - 00 3c 00 2f 00 96 00 41-c0 12 c0 08 00 16 00 13 .<./...A........ 00c0 - 00 10 00 0d c0 0d c0 03-00 0a 00 07 c0 11 c0 07 ................ 00d0 - c0 0c c0 02 00 05 00 04-00 ff 01 00 00 43 00 0b .............C.. 00e0 - 00 04 03 00 01 02 00 0a-00 0a 00 08 00 17 00 19 ................ 00f0 - 00 18 00 16 00 23 00 00-00 0d 00 20 00 1e 06 01 .....#..... .... 0100 - 06 02 06 03 05 01 05 02-05 03 04 01 04 02 04 03 ................ 0110 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01 ................ 0120 - 01 . read from 0x21d3500 [0x21d8ae0] (7 bytes => 0 (0x0)) 140427521087376:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 289 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1558694617 Timeout : 300 (sec) Verify return code: 0 (ok) ---

Thanks everybody for the advice.

Well keeping the number of CDRs left on the PBX low is definitely something we can accomplish. Is there a way to have the PBX clean up old recordings (e.g. remove recordings older than 48 hours). I already copy them to an ftp server anyway so I don't mind.

Thanks! Upgrading a production system is always tough but "a man's got to do...". Any ideas about my other 2 questions: Is the 5 minutes to restart normal? How could we cut it down? Is it OK that I'm running 64bit CentOS but pbxctrl reports CentOS32? 

We had problems with calls not connecting or getting no audio today and when I examined the logs I show many serious errors like these: [1] ... Could not open cdr....bin for writing [1] ... Port ...: Could not allocate sockets! [0] ... UDP(IPv4): Could not open socket (EMFILE) lsof | grep ^pbxctrl | wc -l showed more that 8160 lines. Looking at the output of lsof I noticed that 99% of the lines are like this: COMMAND PID TID USER FD TYPE DEVICE SIZE/OFF NODE NAME pbxctrl 1280 1307 root 1005u IPv6 10107481 0t0 UDP *:49689 I wonder if so many open files(UDP sockets) is normal. It doesn't seem to be so I restarted pbxctrl and the problems went away (after a painful 5 minutes). I'm monitoring the output of `lsof | grep ^pbxctrl | wc -l` and it shows me anywhere from 1200 to 1500 open files (with moderate traffic). I've raised my soft & hard limit of open files from 1024 & 4096 to 4000 & 8000 with prlimit -n -p`pidof pbxctrl` --nofile=4000:8000 Some basic facts about my system: Software-Version: 60.0.3 (CentOS32) Build Date: May 1 2018 04:42:47 uname -a Linux ... x86_64 x86_64 x86_64 GNU/Linux Questions: Have we hit a bug or is such an amount of open sockets normal? If it's normal what's the recommended configuration for CentOS? Is the 5 minutes to restart normal? How could we cut it down? Is it OK that I'm running 64bit CentOS but pbxctrl reports CentOS32?

Thanks for the beta. Unfortunately we didn't get it right with the first try. Here are the results of our tests. We are sending more details and traces in the support mail. We observe that both forwarded and non-forwarded calls have 2 levels of history-info. This is definetely not OK. In the case of the non forwarded call as stated above our peer agrements mandate that we transmit no history-info at all. In the case of the forwarded call where A calls B and is forwarded to C all was fine in v60. However in v61.1 we have a regression: the history-info is still OK but the P-Asserted-Identity field is wrong: it has the B-number instead of the A-number.

We frequently need to find the domain of a DID . Before v60 we would open the DID management and search within the page for the DID (using the built in browser search). But since v60 this is not possible because it presents the DIDs separated in multiple pages. In our case we have far too many pages to visit them all one-by-one. Could you please add a search function in DID Management.