Jump to content
Vodia PBX forum

ndemou

Members
  • Content Count

    165
  • Joined

  • Last visited

Community Reputation

0 Neutral

About ndemou

  • Rank
    Advanced Member

Recent Profile Visitors

8,620 profile views
  1. First of all many thanks for the help Now, regarding the correlation between the output of lsof and the contents of /proc/`pidof pbxctrl`/fd: No they don't match and the difference is clearly on the IP sockets (see [3]).The reason for the difference is that lsof gives 7 lines of output for each socket. So if I count only unique socket ids then both outputs match: $ lsof -nP | grep 'pbxctrl' | grep 'IPv' | awk -F' ' '{print $7 }' | sort -n | uniq | wc -l 116 $ ls -l /proc/`pidof pbxctrl`/fd | grep 'socket' | wc -l 116 A quick check[1] on my saved lsof outputs during the past few weeks reveals that this factor (x7) is constant. And another check [2] reveals that: The unique socket ids at the output of lsof (that match the number of sockets in /proc/.../fd) are increasing with the pattern mentioned in my previous post (i.e. ~4% increase during working days and no increase during weekends and holidays). By the way: when you exclude sockets the number of "files" left is more-or-less constant during these weeks that I'm testing (97 open "files" with a rare peak to at most 137 now and then). Here's a plot of unique socket ids per hour for two weeks. Weekends are colored green (it's of-course the same as above but scaled down by a factor of 7 but I like pictures :-) ______________________________ [1] ls lsof.2019-0*|sort | while read FILE; do cat $FILE | grep 'IPv' | awk -F' ' '{print $7 }' | grep -v '0t0' | sort -n | uniq -c | awk -F' ' '{print $1 }' | uniq -c | sort -nr | head -1; done [2] ls lsof.2019-0*|sort | while read FILE; do echo -n "$FILE "; cat $FILE | grep 'IPv' | awk -F' ' '{print $7 }' | grep -v '0t0' | sort -n | uniq | wc -l; done [3] $ ls -l /proc/`pidof pbxctrl`/fd | grep 'socket'|wc -l 125 $ lsof -nP | grep 'pbxctrl' | grep 'IPv'|wc -l 992
  2. I'm logging the output of lsof for the pbxctrl process every hour for about a month. It is clear that the number of lines of output grows by about 20% per week. I've just restarted pbxctrl as a precaution. I'm attaching a graph. The y-axis is the number of lines of the output of the command: lsof | grep pbxctrl . Weekends and holidays are painted in green color. Most of my customers are businesses so I have many calls with a peak at about midday during working days and very few calls during weekends and holidays. I confess I haven't dived into the details of the output of lsof. However I'm almost sure that no matter the exact meaning it doesn't seem right for the number of these lines to grow by ~4% every working day. Most likely a bug in pbxctrl and less likely one in lsof. I need your help for further debugging though. I have kept all the log files and the full output of lsof for this period. Anything else I can do?
  3. Hmmm good thinking. I'll have to read a bit more on lsof and linux internals. Will let you know
  4. They certainly belong to the same process: pbxctrl with PID 1486. But that's to be expected since I'm looking specifically for pbxctrl. Not sure what you're asking here so I'll provide anything I can think of: "man lsof" reads: "TID is the task (thread) IDentification number". In the example output above the 3rd column is the TID column (empty for the first line and 1530 to 1536 for the next 7 lines). B.T.W. I see the same pattern for almost all other ports: each port appears 8 times, once for the main process and another 7 times for threads. I've restarted pbxctrl and now I'm running lsof -nP | grep pbxctrl every hour and logging the full output on a series of timestamped files. Just in case it helps.
  5. We have a few hundred extensions in small companies. Nothing unusual I would guess (some inter-domain, plenty external calls and a conference call from time to time). Surely no multicast paging. Regarding attacks, I've run for a few minutes ngrep (a packet sniffer) looking for SIP replies that are not "SIP 200 OK" and I see nothing unusual. Regarding IPv6 ports in lsof's output they account for1352 out of 3300 lines. But I should note that they contain only 169 unique port numbers. E.g.: lsof -n| grep ^pbxctrl | grep -i ipv6 |grep 49460 pbxctrl 1486 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1530 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1531 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1532 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1533 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1534 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1535 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1536 root 456u IPv6 18445566 0t0 UDP *:49460 Should I send you log files or anything else?
  6. Unfortunately within a week 'lsof -n | grep pbxctrl' did give more than 4000 lines of output. Usage of the system is stable so I don't like that this number keeps climbing up week by week and I'm afraid of the system crashing again. Looking at today's graphs I see these PEAKS: ~770 Registrations+Subscriptions, 52 call legs 112 http(s) 112. ~ 10 for TFTP & LDAP together In case it helps: Examining the output of ngrep on port 5060 I see that I'm sending packets to 310 IP:port couples. The output of lsof | grep pbxctrl is along the lines of this: pbxctrl 1486 1536 root 442u IPv4 18445564 0t0 UDP *:49460 pbxctrl 1486 1536 root 443u IPv4 18437010 0t0 TCP 1.2.3.4:https->9.10.11.12:53054 (ESTABLISHED) pbxctrl 1486 1536 root 444u IPv4 18445565 0t0 UDP *:49461 pbxctrl 1486 1536 root 452u IPv6 18472707 0t0 UDP *:49869 pbxctrl 1486 1536 root 453u IPv4 18506734 0t0 UDP *:62612 pbxctrl 1486 1536 root 456u IPv6 18445566 0t0 UDP *:49460 pbxctrl 1486 1536 root 457u IPv6 18445567 0t0 UDP *:49461 pbxctrl 1486 1536 root 458u IPv4 18472815 0t0 TCP 1.2.3.4:https->5.6.7.8:34917 (ESTABLISHED) pbxctrl 1486 1536 root 461u IPv4 18506735 0t0 UDP *:62613 pbxctrl 1486 1536 root 472u IPv4 18505204 0t0 UDP *:52692 pbxctrl 1486 1536 root 473u IPv4 18505205 0t0 UDP *:52693 pbxctrl 1486 1536 root 478u IPv4 18505208 0t0 UDP *:52192 pbxctrl 1486 1536 root 479u IPv4 18505209 0t0 UDP *:52193 pbxctrl 1486 1536 root 482u IPv4 18505038 0t0 UDP *:58870 pbxctrl 1486 1536 root 483u IPv4 18505039 0t0 UDP *:58871 pbxctrl 1486 1536 root 484u IPv4 18505042 0t0 UDP *:61862 pbxctrl 1486 1536 root 485u IPv4 18505043 0t0 UDP *:61863 What's your opinion? Would you like more data?
  7. I've upgraded to 63.0.1 about 12 days ago and the output of lsof -n | grep pbxctrl has grown from about ~2000 lines to about ~3300 lines (IPv4 and IPv6 connections). I've set a new warning level at 4000 lines and will let you know if/when I'll hit it
  8. Thanks that was it! I had to temporarily firewall one IP with a lot of TCP connections in order to let me connect to the webUI.
  9. Not quite fixed after all. It works for a little while then it renders pages with huge logos and unusual layout, then after trying ctrl-F5 (firefox's shortcut to *fully* reload) it stops responding. I'm seeing these lines at the logs: [6] 20190525003418: Last message repeated 3 times [3] 20190525003418: Current number of requests 50 has reached maximum 50, connections not accepted [6] 20190525003418: 140 more requests pending to acme-v02.api.letsencrypt.org:443 [6] 20190525003419: Last message repeated 2 times [3] 20190525003419: Current number of requests 50 has reached maximum 50, connections not accepted [6] 20190525003420: 140 more requests pending to acme-v02.api.letsencrypt.org:443 [6] 20190525003422: Last message repeated 5 times [3] 20190525003422: Current number of requests 50 has reached maximum 50, connections not accepted [6] 20190525003422: 140 more requests pending to acme-v02.api.letsencrypt.org:443
  10. FIXED! removed webUI customization (pbxwebai and webpages directory) and it works.
  11. OS is linux/CentOS. There is not firewall (wget ...localhost...). The ports _are_ open (notice that wget reports that it gets connected but I also verified with ss). rawlogin has exactly the same behavior (I get connected but get zero bytes from the PBX). I rebooted (it's deep in the night here) with no success.
  12. Had no problem in test server but after upgrading our production system to 63.0 we lost access to the web interface! Tried from various IPs, from same LAN and from localhost with no success. Ports 8080 and 443 are open and LISTENing, we get connected but then the pbxctrl gives zero bytes back. I'm providing the ouput of tests from localhost with wget and openssl below. Please help as soon as possible. wget -vvvv http://localhost:8080/ --2019-05-24 13:40:19-- http://localhost:8080/ Resolving localhost (localhost)... ::1, ::1, 127.0.0.1 Connecting to localhost (localhost)|::1|:8080... connected. HTTP request sent, awaiting response... No data received. Retrying. [...same error again and again...] wget -vvvv https://localhost --2019-05-24 13:28:21-- https://localhost/ Resolving localhost (localhost)... ::1, ::1, 127.0.0.1 Connecting to localhost (localhost)|::1|:443... connected. Unable to establish SSL connection. openssl s_client -debug -host localhost -port 443 CONNECTED(00000003) write to 0x21d3500 [0x21d3580] (289 bytes => 289 (0x121)) 0000 - 16 03 01 01 1c 01 00 01-18 03 03 58 2a 59 35 05 ...........X*Y5. 0010 - 31 9e c9 3a 58 b9 82 80-ad 03 9c ee cf 4b 2a 1a 1..:X........K*. 0020 - ed 50 8c 11 cf 2b 4a 98-bd be 24 00 00 ac c0 30 .P...+J...$....0 0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1 .,.(.$.......... 0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37 ...k.j.i.h.9.8.7 0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a .6.........2...* 0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../ 0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0 .+.'.#.......... 0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31 ...g.@.?.>.3.2.1 0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43 .0.........E.D.C 00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c .B.1.-.).%...... 00b0 - 00 3c 00 2f 00 96 00 41-c0 12 c0 08 00 16 00 13 .<./...A........ 00c0 - 00 10 00 0d c0 0d c0 03-00 0a 00 07 c0 11 c0 07 ................ 00d0 - c0 0c c0 02 00 05 00 04-00 ff 01 00 00 43 00 0b .............C.. 00e0 - 00 04 03 00 01 02 00 0a-00 0a 00 08 00 17 00 19 ................ 00f0 - 00 18 00 16 00 23 00 00-00 0d 00 20 00 1e 06 01 .....#..... .... 0100 - 06 02 06 03 05 01 05 02-05 03 04 01 04 02 04 03 ................ 0110 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01 ................ 0120 - 01 . read from 0x21d3500 [0x21d8ae0] (7 bytes => 0 (0x0)) 140427521087376:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 289 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1558694617 Timeout : 300 (sec) Verify return code: 0 (ok) ---
  13. Well keeping the number of CDRs left on the PBX low is definitely something we can accomplish. Is there a way to have the PBX clean up old recordings (e.g. remove recordings older than 48 hours). I already copy them to an ftp server anyway so I don't mind.
×
×
  • Create New...