Jump to content

Passwords disappeared


rtl
 Share

Recommended Posts

Last night someone got access to the Vodia management interface on my MT server and caused chaos.

It turned out that the server had wiped the admin password leaving it blank, changed or wiped all the trunk passwords and removed the password restrictions for system admin login. I've seen this before on v66 and v67 (using 67 now) and always managed to catch it before anything like this happened. I've raised it with support and haven't got anywhere. 

It's a real security issue and there seems no explanation so I'm looking for another pbx. Its a shame really as I like Vodia but I can't use it with this enormous security hole.

Link to comment
Share on other sites

An empty admin password on public IP yes is an invitation for chaos. The problem is that for installation we need to start with a well-known default setup, and it does not matter what it exactly is but after the first login the admin should really be forced to change the password. 

I doubt that the password just spontaneously changed. There must have something else happened to cause a password change. Did anybody have write access to the file system? This would be a major problem because yes that can screw up the installation. Please make a copy of the pbx.xml; an empty password for the admin would mean that the pw_pass entry must be empty. 

The other thing that helps is to limit admin access by IP address. We recommend this also for SSH access and it does keep trouble away.

And yes its a good idea to have backups from time to time. If you have a backup, you can go back no matter what happened.

Link to comment
Share on other sites

Right, you don't understand. I'll set the position out clearly and hopefully you will see there's a real issue with security here. 

1. This server has been in operation for 2 years or so.

2. The password we used was secure...32 characters, upper/lowercase/numbers

3. The password has disappeared spontaneously several times. I've raised it with support and the response has been to look for malware on my PC and use incognito mode...as if either of those would make any difference

4. We have access to admin login secured to a few ip addresses. This disappeared too.

5. SIP trunk passwords were changed

6. Users web client passwords were corrupted.

7. We have a backup taken every day

8. SSH access is limited to a single ip address and authenticated by public key. Password login isn't permitted

This isn't the first time it's happened and the only explanation is an issue with your software. This is NOT a new installation.

45 minutes ago, Vodia PBX said:

I doubt that the password just spontaneously changed.

As I have said twice, I seen this several times and raised it with support. The response has been disappointing. As you have not addressed the point at all we have no option other than to find an alternative to Vodia that works as described.

Link to comment
Share on other sites

8 hours ago, RichardDCG said:

Isn't the issue they got access to your MT server?

You just don't understand the post do you. The passwords, admin IP access settings, SIP trunk passwords and user's SIP passwords were set to empty. Thus the server was open to anyone. The issue is why were the passwords deleted? This isn't the first time its happened.

Although yes the issue is indeed they got access to the server but this was due to the Vodia software wiping them. So that is the real issue.

I understand there's an issue whereby when there is a corruption in the CDR exactly this happens...I got this information from Vodia. Vodia have apparently been aware of this for some time but haven't dealt with it. My advice to anyone using or considering Vodia is just don't.

Link to comment
Share on other sites

Of course we take this serious. However we need to understand what has happened here.

If you whitelisted IP access it should be unlikely that someone outside hacked your system. You can check the access log for log entries if you would see any suspicious login activity.

From what I can see the most likely scenario is that your pbx.xml was corrupted and reset to factory after a restart. This would clear the admin account name, the admin password and also the the IP Access settings, and many other things. When the system starts up it needs to figure out if this is a fresh install, and it looks like it came to the conclusion that your system needed that. We have seen cases where the pbx.xml got out of sync, e.g. because of an rsync process overwriting it with similar outcomes. What we have also seen especially on MacOS is that the PBX process starts with without permission to access the file system, and it tries to read the pbx.xml which fails, which will have the PBX initialize it again, and when finally write permissions are granted, it will overwrite the pbx.xml with that new pbx.xml file. This could be a problem e.g. if your firewall blocks the PBX executable after a software update and restart, and a later "Ok" on the firewall admin page. Without being a big Windows expert, I believe that scenario could be happening with the Windows Firewall.

If that is all true, what is the take away? Takeaway #1 is to make a backup before the upgrade, so that if something should go wrong there is a point where you can go back to without headaches. Takeaway #2 is that after a software upgrade you need to tell the firewall somehow that the PBX executable is okay, or at least not ok file system access after the PBX is already running. Takeaway#3 is for us, we'll add a check if the PBX can read a file successfully, and only if that is the case create a new pbx.xml, otherwise exit and not continue the boot up process.

 

Link to comment
Share on other sites

12 hours ago, rtl said:

You just don't understand the post do you.

Maybe not.  But I run Vodia PBX's so if there is an issue I'd like to know about it.  Can you share any other info?  What OS is it on - Windows/Linux/Mac?  I am a bit lost with the MT server - what is that?  Used just as the front end for admin via browser or what the PBX is installed on?  Is this a multi tenant Vodia (we are on 68.0.12 on Linux)?  You've seen it multiple times but I have never seen it, keen to know what differences, if any, we have..

Link to comment
Share on other sites

MT=Multitenant

I'm using debian 10 vodia 67.0.5. Tried v68 just after it was released and it was a bag of nails so we're not going anywhere near it. All I know is what I've described and all the information I have. I've flagged it with Vodia support several times and they're no help. 

I've installed a number of monitoring tools and scripts and can see nothing to point to how this happens. Everything points to an issue with Vodia but without engagement from the company I can't make progress.

In any event we're in the process of moving everyone on the MT server to another pbx. There's a couple of big CPE licences that will follow in time. 

Link to comment
Share on other sites

I've opened several over the last two years and got nowhere apart from trying incognito more and scanning for malware.

I'm moving to another pbx so if I have time I will do so

Link to comment
Share on other sites

So going through the tickets. Actually there was a ticket and it was not a master piece in seeing what the problem was (to put this mildly). What clearly has happened in your system is that the pbx.xml went out of sync with the data for the accounts. This causes the password to be garbled up and in your case it also wiped out the access list for the main admin account. I am sure that for example databases have the same problem if central configuration files are being overwritten. 

Here on the forum we will probably not find out what caused this and its probably too late anyway. In order to prevent this for others in the future there are a few steps that can be done:

  • Before a software upgrade, make a backup. Hard drive space is cheap, and these discussions show how valuable backups are.
  • If you are using file system replication, make sure that the pbx.xml file is in sync with everything else. Overwriting files from the PBX itself and from the replication software will cause chaos.
  • After the initial installation, try rebooting the server and make sure that there is only one PBX instance running. Especially during initial installation it is easy to accidentally start a second PBX which is just sitting there and will grab the HTTP port as soon as the other one ends. A reboot will make sure that this has been cleared.
  • Make sure that nobody has access to the file system unless needed.
Link to comment
Share on other sites

13 hours ago, Vodia PBX said:

This causes the password to be garbled up

It removed the password rather than garbling it.

I've added an .htaccess password to the server.

We're moving to another pbx right now. Our insurance company and lawyers will handle the fallout from this

Link to comment
Share on other sites

On 4/1/2022 at 8:14 AM, Vodia PBX said:
  • Before a software upgrade, make a backup. Hard drive space is cheap, and these discussions show how valuable backups are.

what is the best/recommended way to backup the system?  I manually backup the tenants individually, is there an option to schedule this?   

Link to comment
Share on other sites

I back up each tenant every day using a cron task and I also zip and copy the pbx directory and then zip them all together and send it to S3 storage at digital ocean

Link to comment
Share on other sites

IMHO the ideal backup is a versioning file system where you can move back in time. Some cloud storage providers come with that. My second choice would be a cron job that tar the working directory at 3 AM when there is not much going on, skipping e.g. the recording directory because it just huge.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...