Jon Heese

There are no FXO ports involved here, and as far as I know, star codes do not work on a PRI (and my testing bears this out). However, I will take your advice and contact Sangoma and see if there is any more documentation on exactly how I should go about disabling caller-id. Keep in mind that we're not talking about globally disabling outbound caller ID, but rather selectively disabling it for a specific call, or as a workaround, for a specific extension. Regards, Jon Heese

Any word on this? Regards, Jon Heese

I don't fully understand your first question (or are you talking about "Remote Party/Privacy Indication"?), and I've tried all the possible values in the "Remote Party/Privacy Indication" field, including "No Indication", all with the same results. That's what I meant when I said this: Regards, Jon Heese

Running pbxnsip v3.4.0.3201 (Win32) with a single trunk going to Netborder Express (running on the same server as pbxnsip), which talks to an internal Sangoma A101 single PRI card. Plugged into the Sangoma card we have a PRI from Nuvox Communications. Our issue is with blocking outbound caller ID. Optimally, we'd like to have a "*67" kind of feature, where we could selectively block outbound caller ID on a call-by-call basis, but I understand that the *67 code is a POTS feature, and won't work with a PRI. Instead, we need the switch (i.e. pbxnsip) to generate the proper signal to tell the PRI to block outbound caller ID. So, I'm testing the "Block outbound caller-ID" feature in the extension settings. When I enable that flag and watch the SIP packets, I see that there is an additional "Privacy: id" header added (if I'm using P-Asserted-Identity or P-Preferred-Identity) and an additional "privacy=full" added on to the end of the "Remote-Party-ID" field, if I enable that (I know, I know, it's a legacy option that shouldn't be used). However, the outbound caller ID is NOT being blocked, as it shows up on the callee side as normal. The Netborder logs show the same SIP headers for "privacy" turned on, but it still passes the ANI to the PRI (as it needs to in order for the call to be allowed by our provider). So, my question is: I've got three different players involved here (pbxnsip, Netborder/Sangoma, and Nuvox)... How do I figure out where the problem is? Thanks in advance for any help. Regards, Jon Heese

Okay, how do I turn it off? And will turning it off affect something else negatively? i.e. Is there a reason why it's set this way by default? Also, I don't follow what you mean when you say "the original From/To fields"... Do you mean the fields that show the call as being "To: 305-247-2227"? If so, please understand that this is exactly what we DON'T want. I'm trying to be positive about this, but this has got to be the most unclear, inattentive support I've ever received for a piece of software. I don't even know whether I'm speaking to one person or a bunch of different people. Do you have a real name I can refer to you by? If one of my clients had this issue (and they WILL, once we're confident enough to sell this product), and it took them this long just to explain the problem to tech support, let alone come back with an answer, they would quickly dump me and go with someone else. Regards, Jon Heese

Okay, so consider the following call routing: Incoming PSTN call | v Patton gateway | v Auto-attendant | v Hunt group | v Extension Since the above call came through an auto-attendant, shouldn't it at least show the hunt group as the final destination, not the PSTN/DID number of the company? Regards, Jon Heese

Any word on this? Regards, Jon Heese

That's exactly what I've been trying to tell you! Can you explain this comment a little more? Are you saying that the auto-attendant changes the "To" header to reflect the dialed extension, but the ring group doesn't do this? If so, is there a reason why it behaves this way? Is there any way to make the ring group change the "To" header to properly reflect the extension that picks up the call? It's really quite useless to say that a call was made "To: 305-247-2227", considering that ALL of our incoming calls were made to that number, and what we really want to see is the extension that the caller spoke to. No, we're not using any logging besides the standard "e-mail call logs every night at 12 midnight". No SOAP/CSV/etc. at all. This is purely a call header issue, not something to do with the separate logging mechanism. I can also show this fact by looking at the "Current Calls" page in the web interface. When a call comes in through the ring group, and the caller is speaking with the extension that picked up, you can see in this screen that the "To" header shows "305-247-2227". Regards, Jon Heese

I read through that wiki page a few times and I truly don't understand how it relates to this issue. Just to be clear, we are not having an issue with call routing. Calls are going exactly where we want, when we want. We are having an issue with call *logging*. When an outside caller calls our PBX through our PSTN lines (connected via a 4-port Patton 4114), they are sent directly to an auto-attendant, which allows them to dial an extension or wait to be connected to a ring group. If they dial an extension at that point, they are sent to that extension and the call logs show the caller as being connected to the user whose extension they dialed, like this: Time From To Duration 2009/05/19 09:59:46 COMPANY XYZ ((305)555-4339) Jon Heese (24) 08:07 If they do not dial an extension, and instead they just wait, they are sent to the ring group, and the call logs show a call that looks like this: Time From To Duration 2009/05/19 09:57:34 COMPANY XYZ ((305)555-4339) (305)247-2227 05:36 Both calls above came through the *exact* same PSTN lines, the *exact* same Patton gateway, the *exact* same trunk in pbxnsip, and the *exact* same auto-attendant, and the first place they differ is whether they come through a ring group or go directly to an extension. This is why I think the problem is with the ring group, NOT the gateway or trunk setup. Please confirm that you understand my description above, as I feel like you may be chasing a problem that we don't have. I'm sorry if I seem dense or pedantic, but I just don't understand your logic here. Regards, Jon Heese

Please forgive my skepticism, but I really don't see how this is progress at all. All we've done is change one static field ("anonymous") for another ("305-247-2227"). I just want to make sure that you realize that this number is just our main phone number here in the office, so seeing that all incoming calls (that came through the hunt group) in the logs spoke with "305-247-2227" is really quite uninformative. It seems to me that we're barking up the wrong tree, since the problem only occurs when the call comes through the hunt group. I don't quite follow how that could be an issue with our gateway configuration, since all of the calls come through the gateway the same way, but some calls are logged properly (direct extension dials) and some are not (hunt group calls). Anyway, here are the INVITE messages that show up in the logs on an incoming call: [7] 2009/05/19 10:59:50: SIP Rx udp:192.168.10.240:5060: INVITE sip:3052472227@192.168.10.2 SIP/2.0 Via: SIP/2.0/UDP 192.168.10.240:5060;branch=z9hG4bKf5a01bcd5 Max-Forwards: 70 Content-Length: 224 To: sip:3052472227@192.168.10.2 From: Elite Security <sip:3052455266@192.168.10.2>;tag=ec6505bcbf825e7 Call-ID: ffadce9d8dd6cdf4c231441a0e045cb6@192.168.10.2 CSeq: 597804336 INVITE Route: <sip:192.168.10.2;lr> Supported: timer Remote-Party-ID: Elite Security <sip:3052455266@192.168.10.2>;party=calling;screen=no;privacy=off Content-Type: application/sdp Contact: sip:3052455266@192.168.10.240:5060 Supported: replaces User-Agent: Patton SN4114 JO EUI MxSF v3.2.8.45 00A0BA03DC02 R4.2 2008-03-11 H323 SIP FXS FXO v=0 o=MxSIP 0 147 IN IP4 192.168.10.240 s=SIP Call c=IN IP4 192.168.10.240 t=0 0 m=audio 5158 RTP/AVP 0 8 101 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-15 a=sendrecv [7] 2009/05/19 10:59:50: SIP Tx udp:192.168.10.240:5060: SIP/2.0 100 Trying Via: SIP/2.0/UDP 192.168.10.240:5060;branch=z9hG4bKf5a01bcd5 From: Elite Security <sip:3052455266@192.168.10.2>;tag=ec6505bcbf825e7 To: <sip:3052472227@192.168.10.2>;tag=f47b8af437 Call-ID: ffadce9d8dd6cdf4c231441a0e045cb6@192.168.10.2 CSeq: 597804336 INVITE Content-Length: 0 [5] 2009/05/19 10:59:50: Trunk Patton SN4114 (global) sends call to account 80 in domain st2.st-aubin.com [7] 2009/05/19 10:59:50: SIP Tx udp:192.168.10.240:5060: SIP/2.0 200 Ok Via: SIP/2.0/UDP 192.168.10.240:5060;branch=z9hG4bKf5a01bcd5 From: Elite Security <sip:3052455266@192.168.10.2>;tag=ec6505bcbf825e7 To: <sip:3052472227@192.168.10.2>;tag=f47b8af437 Call-ID: ffadce9d8dd6cdf4c231441a0e045cb6@192.168.10.2 CSeq: 597804336 INVITE Contact: <sip:3052472227@192.168.10.2:5060;transport=udp> Supported: 100rel, replaces, norefersub Allow-Events: refer Allow: INVITE, ACK, CANCEL, BYE, REFER, PRACK, INFO, UPDATE Accept: application/sdp User-Agent: pbxnsip-PBX/3.3.1.3177 Content-Type: application/sdp Content-Length: 216 v=0 o=- 52396 52396 IN IP4 192.168.10.2 s=- c=IN IP4 192.168.10.2 t=0 0 m=audio 55578 RTP/AVP 0 8 101 a=rtpmap:0 pcmu/8000 a=rtpmap:8 pcma/8000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-16 a=sendrecv [7] 2009/05/19 10:59:50: Last message repeated 2 times [7] 2009/05/19 10:59:50: SIP Rx udp:192.168.10.240:5060: ACK sip:3052472227@192.168.10.2:5060;transport=udp SIP/2.0 Via: SIP/2.0/UDP 192.168.10.240:5060;branch=z9hG4bKd689046f8 Max-Forwards: 70 Content-Length: 0 To: sip:3052472227@192.168.10.2;tag=f47b8af437 From: Elite Security <sip:3052455266@192.168.10.2>;tag=ec6505bcbf825e7 Call-ID: ffadce9d8dd6cdf4c231441a0e045cb6@192.168.10.2 CSeq: 597804336 ACK Contact: sip:3052455266@192.168.10.240:5060 User-Agent: Patton SN4114 JO EUI MxSF v3.2.8.45 00A0BA03DC02 R4.2 2008-03-11 H323 SIP FXS FXO Thanks for your help so far. Regards, Jon Heese

Any word on this issue yet? Regards, Jon Heese

Okay, I figured out how to change the "To:" header on incoming calls to the gateway, and I set it to the main number, 3052472227 (all the lines go through the same SIP interface, so there was only one place to set it and they all take that setting). However, all that did was change the "To:" header to "(305)247-2227", instead of "anonymous". Calls that have come through the hunt group still don't show who they spoke with, they just show "(305)247-2227". If the caller dials an extension directly from the AA, that extension does show up in the logs. I can post log snippets if you need to see the difference between the two call strategies. Any other ideas? Regards, Jon Heese

Okay, so any idea how to do any of this in the Patton interface? Regards, Jon Heese

Okay, I understand, but I have no idea how to do what you're describing. We have 4 lines coming in to the Patton SN-4114 gateway and all of them ring in a rollover pattern based on one incoming number, 305-247-2227. Do you know where in the Patton web interface I'm supposed to configure the ports to identify with this number? You understand what I mean when I say that I'm worried this will just replace the "anonymous" in our call logs with our company's phone number, right? Still quite useless. Regards, Jon Heese

Thanks for your reply! Hmmm... The gateway is a Patton SN-4114, and I have to be honest: I have no earthly idea what I'm doing in its web interface 90% of the time. There is nothing intuitive about it, and I feel like I should have gone through a 9 month training class before even touching it. The only way I've been able to configure it correctly in the past is with step-by-step guides in the wiki, etc. I have poked around there for a few minutes just now, but I don't see anything jumping out at me. Something else bothers me about this a little: If this issue is being caused by a setting in the gateway, through which all calls come in, then why is the issue only occurring with calls that come through the Hunt Group, and not calls that are directed to an extension directly? Is there some fundamental difference with the way the call is handled that causes pbxnsip to change the "To" header for direct extension dialed calls and not Hunt Group calls? Furthermore, wouldn't changing some setting in the gateway to show the DID number of the incoming lines (instead of "anonymous") just change our To header from "anonymous" to "305-247-2227"? That's no good either. All we want is for the system to properly modify that header upon transfer out of the Hunt Group and into the individual user that picks up the call. Is this possible? Any further suggestions are greatly appreciated. Regards, Jon Heese

Running pbxnsip v3.3.1.3177. We noticed that incoming calls that come through our main hunt group show up in the calls as being "to" anonymous, even though they connected and spoke with an extension via the hunt group. This is quite annoying for us, because when looking at the call logs, we can't see exactly who an incoming caller spoke with, unless they dialed that person's extension directly, which is rare. Is this intended behavior? A bug? Any way for us to change it to log the final connected extension as the recipient, instead of the "anonymous" hunt group? I set the log level to 7, enabled sip logging and placed a couple test calls, one via the hunt group and one via direct extension dial in the AA. Sure enough, the two log snippets differ in the fact that the hunt group call shows <sip:anonymous@192.168.10.2;user=phone>" in the To: header the whole time, where the direct extension call shows "To: 'Jon Heese' <sip:24@st2.st-aubin.com>" starting at the point where the caller dials my extension. The "From-Header" field on the hunt group is set to "Calling-Party", which I believe is the default. Plus it seems like we're having an issue with the "To:" header, not the "From:". You can see the issue in the attached log files, the message where they start to differ is on line 80 of both. You can see that the "Contact" field is set to <sip:24@72.17.170.196:5060;transport=udp> in both, but the To: header is showing "anonymous" on the hunt group call. Any ideas? Thanks in advance for any assistance you can provide. Regards, Jon Heese call_to_extension.txt call_to_hunt_group.txt

Aha! We already had the country code set to 1, but changing the "Rewrite global numbers" field to "For NANPA (11 digits)" fixed this issue. Thanks for the tip. Regards, Jon Heese

Okay, we recently upgraded from v3.2.x (I forget the exact version) to the 3.3.1.3177 executable, and everything seemed fine, until I tried to dial a long-distance number... Our dial plan is set up like this: The general idea is to make it easy for our users that are used to an old "dial 9 first" system, as well as to make 7-digit dialing possible for numbers in our area's main area code, 305. Then out-of-area dialing is handled. Our trunk is a 4-port Patton FXO box, with 4 PSTN lines plugged into it. Those lines require 1+areacode+number when dialing out-of-area numbers, and areacode+number for local (305 & 786) calls. Local numbers dial fine, with the 7-digit dialing, 786-xxx-xxxx and 305-xxx-xxxx dialing rules (and probably the 9+<local number>, I didn't check), but when I dial 1+areacode+number, I get a tritone error message about how I have to dial 1 before the number... But my dial plan IS dialing 1 before the number... Examples: I dial 18005551212, the dial plan should be following the "150" pref rule, and sending 18005551212 to the trunk, but I get the tritone telling me to dial 1 first. I dial 8005551212, the dial plan should be following the "160" pref rule (those are 10 x's), and sending 1800551212 to the trunk, but I get the tritone telling me to dial 1 first. So it's acting like it's stripping the 1 off the front of the number before it sends it to the trunk (Patton), yet the logs show that it's being sent with the 1... I tried resetting the config on my Patton gateway, with no change. I also tried a little two-stage dialing, using my "230" pref rule, which sends a *67 (caller ID block) to the trunk and then hands me dial tone on the trunk, and I can dial 18005551212 (or whatever) fine and it successfully dials and connects. So then I tried rolling back to the old (3.2.x) pbxctrl.exe executable, and voila, now it's working perfectly. So it's definitely something that changed in the dial plan handling code of the new version. Anyone know what could be causing this? How to nail down my problem? What I can do to fix it? Thanks in advance for any help. Regards, Jon Heese

Right. In a sense, we are lucky that this attack was "lightweight" enough that it isn't using up all of our T1's bandwidth, and all it did was crash the PBX. We are probably going to be asking our T1 provider to add a null route for this IP at the last hop on their side, just to make sure that packets aren't needlessly using up a portion of our bandwidth. Thanks for the insight. Regards, Jon Heese

I agree wholeheartedly. We were down for almost a full 24 hours chasing down .NET updates, license key problems, trunk issues, etc.. If this had been a client's system, instead of our own internal PBX, we would have probably lost the client, or at the very least had a lot of explaining to do as to why neither we nor the system were equipped to handle this situation more quickly. I don't blame the PBX or its developers though. A DoS is a DoS, and I probably should have recognized it earlier than I did. However, if the PBX was made smarter than me (i.e. with a flow control or auto-blacklisting function), then it could have saved my butt even if I was unaware what a UDP packet even was... I talked with Pradeep on the phone a little while ago, and he agrees that it should be possible to implement a more automatic way to mitigate this problem in future versions, so here's to hoping that makes it down the chain and into the software soon! He also mentioned something I hadn't thought of yet in regards to the CS410, which is the possibility of running tcpdump on it via SSH, to determine if a similar symptom is being caused by a packet flood like this. Of course, considering that the CS410 undoubtedly runs on a single-core CPU, that may be more easily said than done. Of course, an enterprise-level router/switch would be able to analyze the packets coming in to the device, but most of our clients are quite small businesses (1-15 employees), so we would probably just pull out a hub and a laptop and run Wireshark there in that kind of situation. Regards, Jon Heese

Okay, well, after looking at a Wireshark capture from this box, I saw a bunch of packets from an IP in Amsterdam, so I tried disabling the WAN adapter on that server, and voila, the CPU usage went back down to normal and everything was fine. So, I blocked that IP address in the "Access" tab of the PBXnSIP config, re-enabled the WAN adapter and now everything's back up and running. It appears that someone was hammering the PBX with approximately 250 SIP "REGISTER" packets per second, thus crippling the process with a bunch of failed login attempts. Of course, when the system had no license installed, it summarily dismissed all incoming SIP registrations, so the problem didn't rear its ugly head until we inserted a license key and it had to process each failed login. Is this not a common occurrence? It seemed to be quite easy for this attacker to bring the system to its knees. Is the prescribed fix for this to do what we did, and block the attacking IP address with the "Access" feature? What if we have this problem on one of the PBXnSIP appliances, like the cs410? We can't do a Wireshark capture on the device, so would we have to depend on the logging or hooking up a test computer with a hub to detect these malicious SIP packets? What would your suggestion be in that case? Thanks for all who helped troubleshoot this and gave ideas. Regards, Jon Heese

Thanks for the key, Matt. It does the same thing. Yes, I tried installing on a new machine, and of course, it runs with the 3-minute license fine, but I can't insert my NFR licenses into it because they are keyed to the MAC address of this server's NIC. It is a dual core server, so I can log in and watch it. The server responds to ping and other network traffic perfectly fine, the only problem is PBXnSIP, which maxes out 1 core of the CPU and totally stops responding to SIP, web, and service control requests. After killing the process, removing the license key from the pbx.xml file, and starting the service back up again, it responds fine (if I don't remove the license from pbx.xml, it will spike out immediately upon starting the service). I'd say there is a 100% chance that it's the application. I'm not really sure what you mean by that last sentence, but Pradeep mentioned a possible bad USB driver in an e-mail to me a little while ago. There are no devices plugged into the server (USB or otherwise), and virtually nothing extraneous connected to it at all. It's just a Dell PowerEdge SC440 server with a network cable and a power cable plugged into it. No expansion cards at all, just a bare stable machine (been installed and running for 2-3 years) with PBXnSIP on it for the last month or two. Regards, Jon Heese

Easier said that done, apparently... I tried requesting a 3-minute trial key (http://www.pbxnsip.com/sales/trial.php) once last night and once again this morning and haven't gotten anything yet... Is this supposed to be an automated process, or am I waiting for a human being to see my request and provide a key? I can just use any 3-minute key, right? Anyone have one handy they can send to me for some quick testing? This is getting kind of desperate; is there any quicker way to have this addressed by support staff? Incidentally, I don't think the key itself is the problem, since we have two different NFR keys for this machine, and both of them cause this issue. I suspect that if/when I get a 3-minute key and try to insert it, it will do the same thing. I think it's something on this machine that is causing the licensing mechanism to freak out when a key is entered. Regards, Jon Heese

Yes, it seems that we apparently followed you over here to PBXnSIP, Matt. I just tried a couple different licenses on this machine, both commercial and free, with the same results. This license doesn't work on another machine (presumably because it's inherently linked to the MAC address of this server's NIC), but it just says "No License" on the other machine, it doesn't lock up like this, so it seems to be something particular to this server, not the license itself (which has worked fine for at least a month before today). Thanks for the suggestions. I can't say for sure that someone didn't leave a PAC open somewhere, but I changed the web port to a different number and restarted the service (which should have rendered any running PAC client disconnected), with the same results. So I don't think it's the PAC. But thanks anyway. Now there's an interesting suggestion... I do know that some Microsoft Updates had been performed on this machine recently (although not since last Friday), and sure enough, Add/Remove Programs does show .NET 3.0 and .NET 3.5 with a handful of updates on both. However, after uninstalling both .NET 3.0 and .NET 3.5 entirely, and rebooting the server, the service still pegs out when started. I'm not sure I understand what you're saying here, but interestingly, as you seem to imply here, only 1 of the 2 processor cores is actually pegging out. Does this shed any light on what might be happening? Thanks again for all the great suggestions, and please keep 'em coming! We are dead in the water! Regards, Jon Heese

Today, around 3:00pm, our internal PBXnSIP install (on a Windows 2003 server) came screeching to a halt. It was maxing out the CPU and was unresponsive to everything: web, SIP, service control, etc. I had to kill the pbxctrl.exe process to get the service to stop, but as soon as I started it back up again, it would go back to max CPU and become unresponsive again within 10-20 seconds, never to return. A reboot of the server made no difference, a restore from known-good tar backup made no difference, upgrading the pbxctrl.exe to 3.3.1.3177 made no difference, but I noticed that a "factory reset" (if I was quick enough to get in the web interface and do it before it locked up) did cause it to behave normally, until I tried restoring from the backup, at which point it went back to the max CPU. I noticed that if I manually remove the license key from pbx.xml, everything starts up fine. So, by my rough estimation, it's either the licensing component itself that's causing the problem, or it's something that is running *because* of the license (i.e. trunk registration, etc.) that's causing it. Either way, we are dead in the water until we can work out what's maxing out the CPU and killing the PBX. Any suggestions? Regards, Jon Heese