Jump to content
Vodia PBX forum

ndemou

Members
  • Content Count

    162
  • Joined

  • Last visited

Community Reputation

0 Neutral

About ndemou

  • Rank
    Advanced Member

Recent Profile Visitors

8,587 profile views
  1. Hmmm good thinking. I'll have to read a bit more on lsof and linux internals. Will let you know
  2. They certainly belong to the same process: pbxctrl with PID 1486. But that's to be expected since I'm looking specifically for pbxctrl. Not sure what you're asking here so I'll provide anything I can think of: "man lsof" reads: "TID is the task (thread) IDentification number". In the example output above the 3rd column is the TID column (empty for the first line and 1530 to 1536 for the next 7 lines). B.T.W. I see the same pattern for almost all other ports: each port appears 8 times, once for the main process and another 7 times for threads. I've restarted pbxctrl and now I'm running lsof -nP | grep pbxctrl every hour and logging the full output on a series of timestamped files. Just in case it helps.
  3. We have a few hundred extensions in small companies. Nothing unusual I would guess (some inter-domain, plenty external calls and a conference call from time to time). Surely no multicast paging. Regarding attacks, I've run for a few minutes ngrep (a packet sniffer) looking for SIP replies that are not "SIP 200 OK" and I see nothing unusual. Regarding IPv6 ports in lsof's output they account for1352 out of 3300 lines. But I should note that they contain only 169 unique port numbers. E.g.: lsof -n| grep ^pbxctrl | grep -i ipv6 |grep 49460 pbxctrl 1486 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1530 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1531 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1532 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1533 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1534 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1535 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1536 root 456u IPv6 18445566 0t0 UDP *:49460 Should I send you log files or anything else?
  4. Unfortunately within a week 'lsof -n | grep pbxctrl' did give more than 4000 lines of output. Usage of the system is stable so I don't like that this number keeps climbing up week by week and I'm afraid of the system crashing again. Looking at today's graphs I see these PEAKS: ~770 Registrations+Subscriptions, 52 call legs 112 http(s) 112. ~ 10 for TFTP & LDAP together In case it helps: Examining the output of ngrep on port 5060 I see that I'm sending packets to 310 IP:port couples. The output of lsof | grep pbxctrl is along the lines of this: pbxctrl 1486 1536 root 442u IPv4 18445564 0t0 UDP *:49460 pbxctrl 1486 1536 root 443u IPv4 18437010 0t0 TCP 1.2.3.4:https->9.10.11.12:53054 (ESTABLISHED) pbxctrl 1486 1536 root 444u IPv4 18445565 0t0 UDP *:49461 pbxctrl 1486 1536 root 452u IPv6 18472707 0t0 UDP *:49869 pbxctrl 1486 1536 root 453u IPv4 18506734 0t0 UDP *:62612 pbxctrl 1486 1536 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1536 root 457u IPv6 18445567 0t0 UDP *:49461 pbxctrl 1486 1536 root 458u IPv4 18472815 0t0 TCP 1.2.3.4:https->5.6.7.8:34917 (ESTABLISHED) pbxctrl 1486 1536 root 461u IPv4 18506735 0t0 UDP *:62613 pbxctrl 1486 1536 root 472u IPv4 18505204 0t0 UDP *:52692 pbxctrl 1486 1536 root 473u IPv4 18505205 0t0 UDP *:52693 pbxctrl 1486 1536 root 478u IPv4 18505208 0t0 UDP *:52192 pbxctrl 1486 1536 root 479u IPv4 18505209 0t0 UDP *:52193 pbxctrl 1486 1536 root 482u IPv4 18505038 0t0 UDP *:58870 pbxctrl 1486 1536 root 483u IPv4 18505039 0t0 UDP *:58871 pbxctrl 1486 1536 root 484u IPv4 18505042 0t0 UDP *:61862 pbxctrl 1486 1536 root 485u IPv4 18505043 0t0 UDP *:61863 What's your opinion? Would you like more data?
  5. I've upgraded to 63.0.1 about 12 days ago and the output of lsof -n | grep pbxctrl has grown from about ~2000 lines to about ~3300 lines (IPv4 and IPv6 connections). I've set a new warning level at 4000 lines and will let you know if/when I'll hit it
  6. Thanks that was it! I had to temporarily firewall one IP with a lot of TCP connections in order to let me connect to the webUI.
  7. Not quite fixed after all. It works for a little while then it renders pages with huge logos and unusual layout, then after trying ctrl-F5 (firefox's shortcut to *fully* reload) it stops responding. I'm seeing these lines at the logs: [6] 20190525003418: Last message repeated 3 times [3] 20190525003418: Current number of requests 50 has reached maximum 50, connections not accepted [6] 20190525003418: 140 more requests pending to acme-v02.api.letsencrypt.org:443 [6] 20190525003419: Last message repeated 2 times [3] 20190525003419: Current number of requests 50 has reached maximum 50, connections not accepted [6] 20190525003420: 140 more requests pending to acme-v02.api.letsencrypt.org:443 [6] 20190525003422: Last message repeated 5 times [3] 20190525003422: Current number of requests 50 has reached maximum 50, connections not accepted [6] 20190525003422: 140 more requests pending to acme-v02.api.letsencrypt.org:443
  8. FIXED! removed webUI customization (pbxwebai and webpages directory) and it works.
  9. OS is linux/CentOS. There is not firewall (wget ...localhost...). The ports _are_ open (notice that wget reports that it gets connected but I also verified with ss). rawlogin has exactly the same behavior (I get connected but get zero bytes from the PBX). I rebooted (it's deep in the night here) with no success.
  10. Had no problem in test server but after upgrading our production system to 63.0 we lost access to the web interface! Tried from various IPs, from same LAN and from localhost with no success. Ports 8080 and 443 are open and LISTENing, we get connected but then the pbxctrl gives zero bytes back. I'm providing the ouput of tests from localhost with wget and openssl below. Please help as soon as possible. wget -vvvv http://localhost:8080/ --2019-05-24 13:40:19-- http://localhost:8080/ Resolving localhost (localhost)... ::1, ::1, 127.0.0.1 Connecting to localhost (localhost)|::1|:8080... connected. HTTP request sent, awaiting response... No data received. Retrying. [...same error again and again...] wget -vvvv https://localhost --2019-05-24 13:28:21-- https://localhost/ Resolving localhost (localhost)... ::1, ::1, 127.0.0.1 Connecting to localhost (localhost)|::1|:443... connected. Unable to establish SSL connection. openssl s_client -debug -host localhost -port 443 CONNECTED(00000003) write to 0x21d3500 [0x21d3580] (289 bytes => 289 (0x121)) 0000 - 16 03 01 01 1c 01 00 01-18 03 03 58 2a 59 35 05 ...........X*Y5. 0010 - 31 9e c9 3a 58 b9 82 80-ad 03 9c ee cf 4b 2a 1a 1..:X........K*. 0020 - ed 50 8c 11 cf 2b 4a 98-bd be 24 00 00 ac c0 30 .P...+J...$....0 0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1 .,.(.$.......... 0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37 ...k.j.i.h.9.8.7 0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a .6.........2...* 0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../ 0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0 .+.'.#.......... 0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31 ...g.@.?.>.3.2.1 0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43 .0.........E.D.C 00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c .B.1.-.).%...... 00b0 - 00 3c 00 2f 00 96 00 41-c0 12 c0 08 00 16 00 13 .<./...A........ 00c0 - 00 10 00 0d c0 0d c0 03-00 0a 00 07 c0 11 c0 07 ................ 00d0 - c0 0c c0 02 00 05 00 04-00 ff 01 00 00 43 00 0b .............C.. 00e0 - 00 04 03 00 01 02 00 0a-00 0a 00 08 00 17 00 19 ................ 00f0 - 00 18 00 16 00 23 00 00-00 0d 00 20 00 1e 06 01 .....#..... .... 0100 - 06 02 06 03 05 01 05 02-05 03 04 01 04 02 04 03 ................ 0110 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01 ................ 0120 - 01 . read from 0x21d3500 [0x21d8ae0] (7 bytes => 0 (0x0)) 140427521087376:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 289 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1558694617 Timeout : 300 (sec) Verify return code: 0 (ok) ---
  11. Well keeping the number of CDRs left on the PBX low is definitely something we can accomplish. Is there a way to have the PBX clean up old recordings (e.g. remove recordings older than 48 hours). I already copy them to an ftp server anyway so I don't mind.
  12. Thanks! Upgrading a production system is always tough but "a man's got to do...". Any ideas about my other 2 questions: Is the 5 minutes to restart normal? How could we cut it down? Is it OK that I'm running 64bit CentOS but pbxctrl reports CentOS32? 
  13. We had problems with calls not connecting or getting no audio today and when I examined the logs I show many serious errors like these: [1] ... Could not open cdr....bin for writing [1] ... Port ...: Could not allocate sockets! [0] ... UDP(IPv4): Could not open socket (EMFILE) lsof | grep ^pbxctrl | wc -l showed more that 8160 lines. Looking at the output of lsof I noticed that 99% of the lines are like this: COMMAND PID TID USER FD TYPE DEVICE SIZE/OFF NODE NAME pbxctrl 1280 1307 root 1005u IPv6 10107481 0t0 UDP *:49689 I wonder if so many open files(UDP sockets) is normal. It doesn't seem to be so I restarted pbxctrl and the problems went away (after a painful 5 minutes). I'm monitoring the output of `lsof | grep ^pbxctrl | wc -l` and it shows me anywhere from 1200 to 1500 open files (with moderate traffic). I've raised my soft & hard limit of open files from 1024 & 4096 to 4000 & 8000 with prlimit -n -p`pidof pbxctrl` --nofile=4000:8000 Some basic facts about my system: Software-Version: 60.0.3 (CentOS32) Build Date: May 1 2018 04:42:47 uname -a Linux ... x86_64 x86_64 x86_64 GNU/Linux Questions: Have we hit a bug or is such an amount of open sockets normal? If it's normal what's the recommended configuration for CentOS? Is the 5 minutes to restart normal? How could we cut it down? Is it OK that I'm running 64bit CentOS but pbxctrl reports CentOS32?
  14. Thanks for the beta. Unfortunately we didn't get it right with the first try. Here are the results of our tests. We are sending more details and traces in the support mail. We observe that both forwarded and non-forwarded calls have 2 levels of history-info. This is definetely not OK. In the case of the non forwarded call as stated above our peer agrements mandate that we transmit no history-info at all. In the case of the forwarded call where A calls B and is forwarded to C all was fine in v60. However in v61.1 we have a regression: the history-info is still OK but the P-Asserted-Identity field is wrong: it has the B-number instead of the A-number.
×
×
  • Create New...