Performance with an increasing number of users

Hey Dion,

Could you please perform the following and send me the results. This will help me diagnosis the issues you are experiencing.

- start RDM with the /Profiler flag (via cmd line) - this will trace RDM start-up times
- now edit any session - this will trace the edit
- hit the "send trace to" button to send us the trace information, please add my name (Stefane) to the message subject

Could you also send us your data source information, I want to make sure your data source settings are optimal. File -> My Data Source Information (top of the form, hit the mail icon)

Hi Stefane,
I've just sent you the information.
Hope you can tell us if the performance could get worse when more users and/or more connections are added.


Kind Regards,
Dion

Dion,

Thank you for the information.

Short answer: with 85+ users using the system, the performance should remain about the same.

Feel free to stop reading here, just jump to the conclusion :-)

--------------------------
Long answer: Most of the processing time in the trace you sent me are being spent on the client not the server. Here are some explanations of what is going on and where time is being spent.

Red - client side
Green - server side (client waiting for server)
Purple - information

Start of RDM - Load data sourceLoadConnections [Microsoft SQL Server/SQL Azure : rdmadm] - 64-bit...[/font]
OfflineManager.GetDataSourceSettings : [color=#cc0000]1837 ms - this is the time it takes to read 3.5MB worth of data from the local offline (SQLite) file[/color]
OfflineManager.LoadConnections...
LoadConnectionsFromOfflineFile...
DeserializeData : [color=#cc0000]3165 ms - create memory objects from the offline data[/color]
ReadFromOffline : 49 ms
LoadConnectionsFromOfflineFile : 11 ms
OfflineManager.LoadConnections : [color=#9900ff]3587 ms - this includes the 3165 ms from above, you can ignore it.[/color]
[color=rgb(153, 0, 255)][font='Times New Roman', serif]Now we have the data from the offline file, we need to check with the server and get the server delta (if any)[/font][/color]
LoadConnectionsFromDataSource...
LoadDataSourceSettingsFromDataSource...
GetDataSourceSettings : [color=#669900]318 ms - fetch settings & session deltas from server[/color]
LoadDataSourceSettingsFromDataSource : 325 ms
OfflineManager.Save : [color=#cc0000]537 ms - save security context to the offline file[/color]
GetConnections...
Database.GetConnections...
Reading connections from database : 0 ms
ConnectionInfos.Foreach...
Decrypt (multi-threaded) : 0 ms
Deserialize (multi-threaded) : 0 ms
ConnectionInfos.Foreach : 3 ms
Database.GetConnections : [color=#cc0000]54 ms - manage and merge any server deltas[/color]
GetConnections : 59 ms
PrepareConnectionList : 0 ms
DoMergeConnections : 5 ms
LoadConnectionsFromDataSource : [color=#9900cc]1298 ms - sub total (325 + 537 + 59 + some non-trace code)[/color]
[font='Times New Roman', serif]LoadConnections [Microsoft SQL Server/SQL Azure : rdmadm] - 64-bit : [/font][color=#9900cc]7246 ms - grand total (1867 + 3587 + 1298 + non-traced)[/color]
[font='Times New Roman', serif]We have good data now, let use display it[/font]
[font='Times New Roman', serif]TreeView.LoadConnectionList...[/font]
GetGroupInfos : 0 ms
BeginUpdate...
BuildGroupList...
FirstLoop : 1 ms
FindConnectionGroup : 39 ms - Count:106
BuildGroupList : 6 ms
ApplyGroupCustomImages : 0 ms
LoadConnectionInTreeNode : 717 ms
PreCreateGroups : 87 ms - Count:1
BeginUpdate : 66 ms
TreeView.LoadConnectionList : [color=#cc0000]874 ms - tree view is now loaded[/color][font='Times New Roman', serif]


So we can see here that not much is going on server side. All the heavy lifting is done on the client.


Refresh/Edit
Database.SaveConnection...

[font='Times New Roman', serif]Database.SaveConnection : [color=#669900]591 ms - save the change[/color][/font]
[font='Times New Roman', serif]LoadConnections [Microsoft SQL Server/SQL Azure : rdmadm] - 64-bit...[/font]
OfflineManager.GetDataSourceSettings : 4 ms
LoadConnectionsFromDataSource...
LoadDataSourceSettingsFromDataSource...
GetDataSourceSettings : [color=#669900]18 ms - fetch server deltas[/color]
LoadDataSourceSettingsFromDataSource : 20 ms
OfflineManager.Save : 13 ms
GetConnections...
Database.GetConnections...
Reading connections from database : 5 ms
ConnectionInfos.Foreach...
Decrypt (multi-threaded) : 4 ms
Deserialize (multi-threaded) : 5 ms
ConnectionInfos.Foreach : 7 ms
Database.GetConnections : 61 ms
GetConnections : 63 ms
PrepareConnectionList : 0 ms
OfflineManager.Save : 224 ms
DoMergeConnections : 293 ms
LoadConnectionsFromDataSource : [color=#cc0000]401 ms - merge changes locally[/color]
LoadConnections [Microsoft SQL Server/SQL Azure : rdmadm] - 64-bit : 485 ms
TreeView.LoadConnectionList...
GetGroupInfos : 0 ms
BeginUpdate...
BuildGroupList...
FirstLoop : 0 ms
FindConnectionGroup : 34 ms - Count:106
BuildGroupList : 7 ms
ApplyGroupCustomImages : 0 ms
LoadConnectionInTreeNode : 367 ms
PreCreateGroups : 114 ms - Count:1
BeginUpdate : 51 ms
TreeView.LoadConnectionList : [color=#cc0000]537 ms - load tree view[/color]
ThreadQueue.AddRow : [color=#cc0000]226 ms - save delta to the offline file - background thread - doesn't affect the client[/color]-----------------------------

Conclusion, your initial load takes just over 8 seconds and an edit takes about 1.5 seconds(actually 3.7, see point #2). I do have a few areas of concern:


1 - DeserializeData : 3165ms - We've done work recently in this. I'm fairly certain that they are not in the v9.0.12. Could you try with the v9.1.0.0 release. It could trim 1+ seconds from this step
2 - PersonalConnectionManager.Refresh : 2264 ms - I will investigate, not sure why the refresh is taking so long and why it's even needed. This could be why your save takes so long.


Sorry for the long explanation. If anything is not clear, please let me know.


Regards,

Hi Stefane,
I downloaded version 9.1 to see the difference.
In this single test, the startup seemed actually slower but ... Yesterday, we also added a bunch of new connections which may cause this. Even my hybrid-ssd disk may be involved. The edit on the other hand, seemed faster.
However, you said that most of the time, it's the workstation/laptop that is busy.
Our concern was, do more users cause more delay? Since this is likely to be false, it's ok for us now.
Thank you for the analysis and the support.


Kind Regards,
Dion




In case you like to see the profiler data;
Start-up:
LoadConnections [Microsoft SQL Server/SQL Azure : myBrand-RDM] - 64-bit...
OfflineManager.GetDataSourceSettings : 655 ms
OfflineManager.LoadConnections...
LoadConnectionsFromOfflineFile...
DeserializeData : 7877 ms - Count : 2191
ReadFromOffline : 68 ms
LoadConnectionsFromOfflineFile : 7890 ms
OfflineManager.LoadConnections : 8454 ms
LoadConnectionsFromDataSource...
LoadDataSourceSettingsFromDataSource...
GetDataSourceSettings : 321 ms
LoadDataSourceSettingsFromDataSource : 332 ms
OfflineManager.Save : 2851 ms
GetConnections...
Database.GetConnections...
Reading connections from database : 0 ms
DecryptData : 2 ms
DeserializeData : 0 ms
Database.GetConnections : 53 ms
GetConnections : 60 ms
PrepareConnectionList : 0 ms
DoMergeConnections : 7 ms
LoadConnectionsFromDataSource : 3930 ms
DecryptData : 3 ms - Count : 3
DeserializeData : 1 ms - Count : 3
LoadConnections [Microsoft SQL Server/SQL Azure : myBrand-RDM] - 64-bit : 13585 ms
TreeView.LoadConnectionList...
GetGroupInfos : 0 ms
BeginUpdate...
ApplyGroupCustomImages : 0 ms
LoadConnectionInTreeNode : 797 ms
PreCreateGroups : 91 ms - Count : 1
BeginUpdate : 49 ms
TreeView.LoadConnectionList : 942 ms
LoadTrayiconContextMenu : 11 ms
LoadConnectionList...
LoadConnectionList : 92 ms
LoadConnectionList...
LoadConnectionList : 36 ms
LoadConnectionList...
LoadConnectionList : 32 ms
LoadConnectionList...
LoadConnectionList : 17 ms
TreeView.LoadConnectionList...
GetGroupInfos : 0 ms
BeginUpdate...
ApplyGroupCustomImages : 0 ms
LoadConnectionInTreeNode : 1 ms
PreCreateGroups : 0 ms
BeginUpdate : 11 ms
TreeView.LoadConnectionList : 23 ms


Edit connection:
Database.SaveConnection...
Database.SaveConnection : 81 ms
LoadConnections [Microsoft SQL Server/SQL Azure : myBrand-RDM] - 64-bit...
OfflineManager.GetDataSourceSettings : 7 ms
LoadConnectionsFromDataSource...
LoadDataSourceSettingsFromDataSource...
GetDataSourceSettings : 31 ms
LoadDataSourceSettingsFromDataSource : 33 ms
OfflineManager.Save : 296 ms
GetConnections...
Database.GetConnections...
Reading connections from database : 0 ms
DecryptData : 0 ms - Count : 1
DeserializeData : 1 ms - Count : 1
Database.GetConnections : 34 ms
GetConnections : 40 ms
PrepareConnectionList : 0 ms
DecryptData : 0 ms - Count : 3
DeserializeData : 3 ms - Count : 3
OfflineManager.Save : 165 ms
DoMergeConnections : 227 ms
LoadConnectionsFromDataSource : 639 ms
DecryptData : 0 ms - Count : 3
DeserializeData : 3 ms - Count : 3
LoadConnections [Microsoft SQL Server/SQL Azure : myBrand-RDM] - 64-bit : 760 ms
TreeView.LoadConnectionList...
GetGroupInfos : 0 ms
BeginUpdate...
ApplyGroupCustomImages : 0 ms
LoadConnectionInTreeNode : 200 ms
PreCreateGroups : 45 ms - Count : 1
BeginUpdate : 29 ms
TreeView.LoadConnectionList : 283 ms
ThreadQueue.AddRow : 34 ms - Count : 1

We are seeing performance degrades also over time with adding more users and sessions. We have about 25 users and 3000 sessions using RDM 8 enterprise MSSQL 2008 and it has been getting slower and slower as we have grown. I thought it was related to the RDM 7 to RDM 8 upgrade with all the rich GUI interface but i was told that should not be the issue.

Are there other performance measurements we analyze with in RDM or is it a one by one submission to forum to diagnose?

Also are there any best practices to settings to optimize?

@Spooner

Please send us a profile trace of the startup & full refresh for analysis along with your data source information.

- start RDM using /Profiler
- once loaded CTRL+refresh (icon)
- click "Send trace to" button
- mention this thread and my name in the subject

- File -> My Data Source Information
- click the mail icon, and send us your information

I will have a look at the numbers to see if there is anything in particular that might be slowing it down.

Regards,

Stefane,
We are using RDM enterprise 8.2 and i did not see the " Send trace to" option so i am just posting back in forum here. We are using MSSQL 2008 R2. Ever since the ribbon interface was introduced it has been running slow. Of course as we have been adding more users it gets even slower. I am testing 9.1.2 version but so far it does not look to improve performance. My team is repeatedly asking me what we can do to improve RDM response time and i am at a loss so far.

Note we are running this in a terminal server environment.



LoadConnectionsFromDataSource...
Database.GetConnections...
Database.GetConnections : 2221 ms
SaveOfflineFile...
SaveOfflineFile : 0 ms
LoadConnectionsFromDataSource : 2595 ms
TreeView.LoadConnectionList...
TreeView.LoadConnectionList : 754 ms


We are contemplating moving up to the manager server but we are questioning if it will make a difference with performance if we are running into issues now with 30 users and 3000 sessions in RDM.

Looking at you trace it takes about 2-1/2 seconds to "load the data" plus 3/4 seconds to "load the tree view". So a total of about 3-1/4 seconds every time "something changes".

Let's define a few terms:
Load the data
Call out to the data source (SQL, MySQL, SQLite...), fetch the sessions & metadata, decrypt & deserialize the session.

Load the tree view
Take all those sessions and display them in the tree view in the proper order, display icons, etc.

Something changes
Any session action (new, modified or delete). So the more users and the more sessions the more this has a chance of happening. In your case both are causing issues. 30+user with 3000+ sessions.


Where are we at with v8.2
Since v7 you can enable client "Caching" (aka: Simple caching in v9) that would only refresh the data once a change was detected. What does that mean? A user clicks edit, the system detects a change, refreshes (takes 3+seconds) and the edit session UI opens. The end result is that the user feels the delay, much like a hang of freeze. If no change was detected then the UI should open within about a second.

What about v9
We've added a new caching mode called "Intelligent cache" to optimize the loading of a large data source. Once the data source is loaded we only ever transfer/load/read deltas of the changes. This affects the "Load the data" part of the trace from 2-1/2 seconds in your case to about 100ms to simple detect that there is no change or something like 300ms when changes are detected and need to be synced. Intelligent cache allows for many more users changing the data while not wreaking havoc on load times for all other users of the system.

What are your options
RDMS (Remote Desktop Manager Server) will not, in your case, help much with performance. From what I've gathered so far your issue are strictly client side. RDMS does add another layer of caching but it's server side.

You can try the old style UI see this topic (Classic mode): http://help.remotedesktopmanager.com/index.html?file_options_ui_ui.htm

Split the large data source into multiple data sources, not always practical, but can lead to faster load times.

Future diagnostic
What is slow? Initial load? Edit a session? Identify your pain points. Now for those pain points start the profiler (Help -> Diagnostics -> Launch profiler) perform the action and send us the trace output including a comment of the action taken and result.

More information can be found here:
http://help.remotedesktopmanager.com/index.html?datasource_caching.htm

Regards,

Please run your tests in v9.x.
edited by stefane on 2/17/2014

Stefane,
I do think the issue is client side i could live with 3.5 second delay but what we are seeing since adding more user is the console is overall slow taking 30-60 seconds to load the console and normal activity like scrolling and expanding tree will show 5-20 delays with periodic white out with not responding in title bar. I will bump more resources on the terminal server but it is shocking how the performance has degraded so much with additional users.

I did run test with 9.1.2 trial client on my laptop and it did not show any slowness but that me accessing a test data source also. I did increase the database refresh on users datasource connections maybe that is issue? People were complaining that they were not seeing updates that is why i increased the database refresh frequency.

Is there a thread that might talk about terminal usage of RDM and maybe have some best advise in that situation?

Wow that is very slow. I see what you mean.

So you have 30+ RDM instances running on a terminal server, each using 100-500MB of RAM (or more depending on # of open sessions) + CPU... ouch

On terminal server I would definitely go with the classic UI, it's less CPU & memory intensive. I would also deploy v9, some of it's features will also help reduce the time (database refresh will use the intelligent cache for example).

On you test laptop, can you access the live data source for testing? Is it possible to have the app deployed locally instead of TS?