Vaults and Offline availability

Hello Thomas,

Which version of RDM are you running?
And what is the data source used?

Regards,

Hi David, we're running 2019.1.30.0 on SQL Server.

Background vault refresh/load feature is on the radar but has no ETA. The way RDM is built the change won't be simple so we need to make sure we do it properly. To do so we need to put a few more things in place before tackling this.

Until then, you can do a few things. Increase the offline expiration days from the default 7 days to say 14 days and then make sure you open/refresh each and every vault every 14 days minimum. This isn't perfect but it will work.

Best regards,

Hi Stefane,

Thank you for the insight. Right now, we have a database with 27000+ entries and editing one entry is soooo slow that a lot of colleagues sometimes ommit documenting something because of it. That is a really big problem for us. We just took over another company and we expect the database to grow to about 35000 entries.

In our test database, I split the connections into 26 vaults (one for each letter of the alphabet) so the workaround is not feasible. Even with only two vaults, the workaround will not be accepted by our 100 engineers who go on-site to customers on a daily basis. Especially since we want to keep using the full treeview (better/easier for the service desk) and this vault thing is already a workaround.

I've noticed that the bottleneck is not getting the data from the database but actually populating the treeview with all the XML data. Could that be correct? So right now, when editing an entry, it refreshes the whole treeview in stead of that single entry. Is there a possibility to have only the edited item be refreshed in the treeview?

Best regards,
Thomas

Wow 27000+ entries that's a huge number of sessions. We usually tell our clients to try to stay below 3000 entries per vault. I can see how, with 26 vaults, managing the offline cache can be a problem.

You are correct, with offline caching, getting the XML from the server is not the problem. We must then decrypt, deseralize, compute the tree hierarchy then display it in the tree view. In the case of an add/edit/delete we must recompute the tree hierarchy and redisplay the information in the tree. I've yet to perform tests with 27000 entries but I can safely assume that this will take some time.

If you want I can analyze your profiler trace of an add to see exactly where time is being spent.

• Help > Profiler
• Uncheck "auto-clear"
• Set "Profiler level" = 1
• Perform an add
• Hit "Send Trace to Support"
• Add "Attn: Stefane" to the subject

What do you think if I add an action (button) where a user can initiate a load/refresh of all vaults? So one click that causes RDM to load the first vault, refresh the sessions, move to the next vault, repeat until all vaults are synchronized to their offline files. Although not perfect but at least you would be synchronized with having to manually cycle through each vaults. Would that help?

Also with 26 vaults, I would create the first vault as my "Index" vault, that has a "Vault Link" to my other 26 vaults and have a "Main Vault" link on top of each vault back to the main vault. You can use "Edit Entry" > "User Interface" > Sort Priority = 1 to force those sessions to the top of the tree view. (see attached video for example) You can also link to a particular folder within a vault.

Best regards,

Yes, I know it's insane but we need all that data. I've already moved all the disabled servers to another vault but the amount is negligible. We're coming up close to 300 customers and all servers, switches, firewalls, storage boxes, passwords, etc. are stored in it.

Thank you for proposing that button but our colleagues will undoubtedly forget to press it once they're at the office. It would be great if you could add an option to trigger it automatically when starting in 'online' mode. That way, it still takes a long time starting up but that's just once a day... When starting up at a customer site or editing an entry at the office, it won't take long at all and our colleagues would really appreciate that.
Maybe an option to make the search box on the bottom of the treeview do a multivault search by default, that would help our service desk colleagues who have to jump from one customer to another.

We actually only use RDM for basic HTTP / SSH / RDP connections and password management and logging of who connects where at what time. Other than the slow editing/adding/starting time, we think RDM is top of the line but I'm still dreaming of a light version (back to basics) with a simplified database.

I'll think about the vault links after I get a proof of concept working like I want it.

I sent you the profiler data b.t.w.

Best regards,
Thomas

Thomas,

I got the profiler logs thank you. Here is what I can gather. Note: I think we can get load times under 20 seconds maybe even 15 seconds for a full load. Changes were made in November 2019 that will help, these changes are in the newest v20.1 and the previous v19.2 version (both will require a database upgrade but are backwards compatible with v19.1).

Here are the highlights of your profiler logs, technical but interesting nonetheless:

1 - Physically getting the data from the server - 8.2 seconds total (1.3s read/send over the network, 2.5s decrypt, 2.5s deserialize). This about as fast as it's going to get for 25,000+ sessions.

GetConnections...
    Database.GetConnections...
        Reading connections from database : 1305 ms
        DecryptData : 252 ms - Count : 25135
        DeserializeData : 6588 ms - Count : 25135
    Database.GetConnections : 8194 ms
GetConnections : 8196 ms

2 - Manipulating the sessions in memory, checking the security... 39.4 seconds total, that should take about 1-2 seconds with the latest build.

DoMergeConnections...
    PrepareConnectionList : 68 ms
    Loop...
        MigrateConnection : 7 ms
        OfflineManager.Merge : 0 ms
        MergeWithList - Contains : 497 ms
        MergeWithList : 37851 ms
    Loop : 38871 ms
    Filter...
        DecryptData : 0 ms
        DeserializeData : 0 ms
    Filter : 377 ms
    ClearConnections : 0 ms
    VerifyMissingAddons : 25 ms
DoMergeConnections : 39444 ms

3 - Load tree view (2 seconds) - this is practically maxed out performance-wise for 25000+ sessions

TreeView.LoadConnectionList...
    GetGroupInfos : 0 ms
    BeginUpdate...
        LoadConnectionInTreeNode : 1611 ms
        PreCreateGroups : 410 ms - Count : 1
    BeginUpdate : 2075 ms
TreeView.LoadConnectionList : 2075 ms

That being said, the profiler trace you sent me is from a "full-load" of the data, if you look at the line below you will see that the hasSecurityChanged is true this will force RDM to reload the entire vault since security has changed and permission on folders needs to be reevaluated. Many actions in RDM will cause the security to change forcing a "full-reload". The main one is changing of Users/Roles in RDM. Try to keep them to a minimum or do them in batches (early AM or late PM) to minimize impact on others using RDM.

reload: False hasSecurityChanged: True hasChanged: True

In the case where hasSecurityChange is false & hasChaged is true, RDM will simply synchronize the changes (add/edit/deletes) and that should take 1-2 seconds max with an additional 2 seconds to reload the tree view.
The last case is both are false, in this case RDM will simply not reload and take less then 1 second. For example:

reload: False hasSecurityChanged: False hasChanged: False
LoadConnections [Microsoft SQL Server : Roles - v14.6.3]...
    ClearConnections : 0 ms
LoadConnections [Microsoft SQL Server : Roles - v14.6.3] : 11 ms

You can do the test on your side, with RDM running start the Profiler (Help > Profiler). You can perform all three scenarios

1. Normal refresh (both false)
   1. 100ms or less
2. CTRL+Refresh button (top left) (both will be true)
   1. 50+ seconds with v19.1, 15'ish seconds with v19.2 or grater
3. Have a colleague change a session and refresh from your machine (false, true)
   1. Should take less than 5 seconds total

Let me know if any of this makes sense. I'm working on adding the "Prompt to sync all vaults" on startup of RDM.

Best regards,

Thank you Stefane!

We seem to always have a full refresh and only change users when a new colleague starts or quits... Maybe something is wrong with our configuration?

I've got a busy schedule for today but I'll be checking out these things tomorrow.

Best regards,
Thomas

FYI I've implemented the "Prompt refresh all vaults on startup" feature we talked about. It should be available in the next release.

How to set it up and use:

1. System Settings > Offline > “Prompt refresh all vaults on startup”
2. Close RDM
3. Open RDM
4. You will be prompted for a full refresh
   1. if not offline
   2. if not disconnected
   3. after you are prompt for credentials/MFA
   4. if you have access to more than 1 vault.

Let me know if that helps.

Also, when you have time we can look at why you are always getting full refreshes, that should not be the case.

Best regards,

Bonjour Stéfane,

J'ai essayé l'option “Prompt refresh all vaults on startup”, mais ça ne fonctionne pas. Nous utilisons l'AD pour la connection, je vois qu'il y a une note pour ce scénario mais ce n'est pas très clair.


Nous avons la version 2020.2.19.0.

Merci

Bonjour Charles,

Merci de nous avoir contacté a ce sujet!

Étant donné que vous utilisez votre Active Directory pour établir la connexion a votre Data Source, cette option ne fonctionnera pas pour vous.

Cordialement,