Passwords disappeared

[rtl]

Last night someone got access to the Vodia management interface on my MT server and caused chaos.

It turned out that the server had wiped the admin password leaving it blank, changed or wiped all the trunk passwords and removed the password restrictions for system admin login. I've seen this before on v66 and v67 (using 67 now) and always managed to catch it before anything like this happened. I've raised it with support and haven't got anywhere. 

It's a real security issue and there seems no explanation so I'm looking for another pbx. Its a shame really as I like Vodia but I can't use it with this enormous security hole.

[Vodia PBX]

An empty admin password on public IP yes is an invitation for chaos. The problem is that for installation we need to start with a well-known default setup, and it does not matter what it exactly is but after the first login the admin should really be forced to change the password. 

I doubt that the password just spontaneously changed. There must have something else happened to cause a password change. Did anybody have write access to the file system? This would be a major problem because yes that can screw up the installation. Please make a copy of the pbx.xml; an empty password for the admin would mean that the pw_pass entry must be empty. 

The other thing that helps is to limit admin access by IP address. We recommend this also for SSH access and it does keep trouble away.

And yes its a good idea to have backups from time to time. If you have a backup, you can go back no matter what happened.

[rtl]

Right, you don't understand. I'll set the position out clearly and hopefully you will see there's a real issue with security here. 

1. This server has been in operation for 2 years or so.

2. The password we used was secure...32 characters, upper/lowercase/numbers

3. The password has disappeared spontaneously several times. I've raised it with support and the response has been to look for malware on my PC and use incognito mode...as if either of those would make any difference

4. We have access to admin login secured to a few ip addresses. This disappeared too.

5. SIP trunk passwords were changed

6. Users web client passwords were corrupted.

7. We have a backup taken every day

8. SSH access is limited to a single ip address and authenticated by public key. Password login isn't permitted

This isn't the first time it's happened and the only explanation is an issue with your software. This is NOT a new installation.

45 minutes ago, Vodia PBX said:

I doubt that the password just spontaneously changed.

As I have said twice, I seen this several times and raised it with support. The response has been disappointing. As you have not addressed the point at all we have no option other than to find an alternative to Vodia that works as described.

[RichardDCG]

12 hours ago, rtl said:

Last night someone got access to the Vodia management interface on my MT server and caused chaos.

 

Isn't the issue they got access to your MT server?

[Vodia Support EU]

20 hours ago, rtl said:

It turned out that the server had wiped the admin password leaving it blank

I am also very interested in the topic. Which server deleted the admin password?

[rtl]

8 hours ago, RichardDCG said:

Isn't the issue they got access to your MT server?

You just don't understand the post do you. The passwords, admin IP access settings, SIP trunk passwords and user's SIP passwords were set to empty. Thus the server was open to anyone. The issue is why were the passwords deleted? This isn't the first time its happened.

Although yes the issue is indeed they got access to the server but this was due to the Vodia software wiping them. So that is the real issue.

I understand there's an issue whereby when there is a corruption in the CDR exactly this happens...I got this information from Vodia. Vodia have apparently been aware of this for some time but haven't dealt with it. My advice to anyone using or considering Vodia is just don't.

[Vodia PBX]

Of course we take this serious. However we need to understand what has happened here.

If you whitelisted IP access it should be unlikely that someone outside hacked your system. You can check the access log for log entries if you would see any suspicious login activity.

From what I can see the most likely scenario is that your pbx.xml was corrupted and reset to factory after a restart. This would clear the admin account name, the admin password and also the the IP Access settings, and many other things. When the system starts up it needs to figure out if this is a fresh install, and it looks like it came to the conclusion that your system needed that. We have seen cases where the pbx.xml got out of sync, e.g. because of an rsync process overwriting it with similar outcomes. What we have also seen especially on MacOS is that the PBX process starts with without permission to access the file system, and it tries to read the pbx.xml which fails, which will have the PBX initialize it again, and when finally write permissions are granted, it will overwrite the pbx.xml with that new pbx.xml file. This could be a problem e.g. if your firewall blocks the PBX executable after a software update and restart, and a later "Ok" on the firewall admin page. Without being a big Windows expert, I believe that scenario could be happening with the Windows Firewall.

If that is all true, what is the take away? Takeaway #1 is to make a backup before the upgrade, so that if something should go wrong there is a point where you can go back to without headaches. Takeaway #2 is that after a software upgrade you need to tell the firewall somehow that the PBX executable is okay, or at least not ok file system access after the PBX is already running. Takeaway#3 is for us, we'll add a check if the PBX can read a file successfully, and only if that is the case create a new pbx.xml, otherwise exit and not continue the boot up process.

 

[RichardDCG]

12 hours ago, rtl said:

You just don't understand the post do you.

Maybe not.  But I run Vodia PBX's so if there is an issue I'd like to know about it.  Can you share any other info?  What OS is it on - Windows/Linux/Mac?  I am a bit lost with the MT server - what is that?  Used just as the front end for admin via browser or what the PBX is installed on?  Is this a multi tenant Vodia (we are on 68.0.12 on Linux)?  You've seen it multiple times but I have never seen it, keen to know what differences, if any, we have..

[rtl]

MT=Multitenant

I'm using debian 10 vodia 67.0.5. Tried v68 just after it was released and it was a bag of nails so we're not going anywhere near it. All I know is what I've described and all the information I have. I've flagged it with Vodia support several times and they're no help. 

I've installed a number of monitoring tools and scripts and can see nothing to point to how this happens. Everything points to an issue with Vodia but without engagement from the company I can't make progress.

In any event we're in the process of moving everyone on the MT server to another pbx. There's a couple of big CPE licences that will follow in time. 

[Vodia PBX]

We still would like to get to the bottom of this. It would be great if you could open a support ticket in the ticket system. 

[rtl]

I've opened several over the last two years and got nowhere apart from trying incognito more and scanning for malware.

I'm moving to another pbx so if I have time I will do so

[Vodia PBX]

So going through the tickets. Actually there was a ticket and it was not a master piece in seeing what the problem was (to put this mildly). What clearly has happened in your system is that the pbx.xml went out of sync with the data for the accounts. This causes the password to be garbled up and in your case it also wiped out the access list for the main admin account. I am sure that for example databases have the same problem if central configuration files are being overwritten. 

Here on the forum we will probably not find out what caused this and its probably too late anyway. In order to prevent this for others in the future there are a few steps that can be done:

• Before a software upgrade, make a backup. Hard drive space is cheap, and these discussions show how valuable backups are.
• If you are using file system replication, make sure that the pbx.xml file is in sync with everything else. Overwriting files from the PBX itself and from the replication software will cause chaos.
• After the initial installation, try rebooting the server and make sure that there is only one PBX instance running. Especially during initial installation it is easy to accidentally start a second PBX which is just sitting there and will grab the HTTP port as soon as the other one ends. A reboot will make sure that this has been cleared.
• Make sure that nobody has access to the file system unless needed.

[rtl]

13 hours ago, Vodia PBX said:

This causes the password to be garbled up

It removed the password rather than garbling it.

I've added an .htaccess password to the server.

We're moving to another pbx right now. Our insurance company and lawyers will handle the fallout from this

[RichardDCG]

On 4/1/2022 at 8:14 AM, Vodia PBX said:

• Before a software upgrade, make a backup. Hard drive space is cheap, and these discussions show how valuable backups are.

what is the best/recommended way to backup the system?  I manually backup the tenants individually, is there an option to schedule this?   

[rtl]

I back up each tenant every day using a cron task and I also zip and copy the pbx directory and then zip them all together and send it to S3 storage at digital ocean

[Vodia PBX]

IMHO the ideal backup is a versioning file system where you can move back in time. Some cloud storage providers come with that. My second choice would be a cron job that tar the working directory at 3 AM when there is not much going on, skipping e.g. the recording directory because it just huge.

[mskenderian]

The PBX should not allow blank password. Reading https://doc.vodia.com/docs/login#admin-password-reset. This is incorrect way to handle password resets. If the PBX cant read that file, then PBX should assume there is a corrupts file, and not allow blank passwords. PBX should write to log file that it cant find the pbx.xml file, and not let any admin users login, until we insert a new MD5(password).

[RichardDCG]

so the issue may not even be Vodia .. although its a bit of a risk being able to reset a password like that.  If I get access to the underlying server then I can change the file at will.  

[mskenderian]

20 hours ago, RichardDCG said:

so the issue may not even be Vodia .. although its a bit of a risk being able to reset a password like that.  If I get access to the underlying server then I can change the file at will.  

well that is expected, if you get access to any underlying server, all bets are off. once someone has access to the servers, they already have everything, all settings, trunks, web passwords.

 

I think the point is that the PBX should look at that config and expect a password, not allow default to be blank. most software these days, auto generate a PW on install and then show it to the user, so they can login. or we need a command part of the install process, to create the main super user.

[Vodia PBX]

On 8/2/2022 at 3:16 PM, mskenderian said:

I think the point is that the PBX should look at that config and expect a password, not allow default to be blank. most software these days, auto generate a PW on install and then show it to the user, so they can login. or we need a command part of the install process, to create the main super user.

There is actually a setting that must be explicitly enabled to allow empty passwords. Believe it or not, but we have customers with devices that cannot have passwords. But that is clearly an exception. If that setting is disabled, empty passwords might be in the account, but you can't do anything with them because it will just not authenticate. Apart from that, when you create e.g. a new extension, you can have the PBX generate a random password that the user usually will never see. 

[mskenderian]

1 minute ago, Vodia PBX said:

There is actually a setting that must be explicitly enabled to allow empty passwords. Believe it or not, but we have customers with devices that cannot have passwords. But that is clearly an exception. If that setting is disabled, empty passwords might be in the account, but you can't do anything with them because it will just not authenticate. Apart from that, when you create e.g. a new extension, you can have the PBX generate a random password that the user usually will never see. 

We are talking about the main admin password, not individual sip accounts. Yes I have seen that setting in the pbx.xml file.

there should never be a instance where the main super admin has a blank password.