How can I improve the performance of Matlab Data Functions in Spotfire/Tibco?

We are starting to use Matlab to write Data Functions called by the Spotfire/Tibco Business Intelligence software instead of using TERR/R/Splus. Both Matlab & Spotfire are installed on a Wintel server with 4 core processors and 8Gb RAM. The performance is EXTREMELY slow. E.g. the Data Function calculate correlations on a matrix or 2552 columns by 20 rows takes 34 minutes, on a matrix of 506 columns by 20 rows takes 2 minutes when one single user is running the process in a local client session (the process can also be run over a Web player). The calculations take seconds when the Data Function is run in Matlab alone on our PCs which have similar hardware configurations. Is this a known limitation of using Matlab and Spotfire together or are adjustments required to the Matlab or Spotfire configuration?. We have been looking at the issue with Tibco support but not been able to improve the situation.

[Michel]

We tried two last things:

1) We monitored thoroughly network traffic on the server and on the client while running the Spotfire dashboards. We did not see any indication of bottleneck or other issue. Bandwith utilizations varied from 20% to 40%

2) We doubled again the hardware to 16Gb RAM and 8 processors (virtual). There is no improvement in performance

At this stage we feel the performance issue lies squarely in the Spotfire to Matlab off-the-shelf functionality

[Michel]

Latest news from Tibco Spotfire: Case 01257893

"Thank you for your update. I [Tibco Spotfire Support] have discussed this issue with our developers, and the consensus is that the transfer of data between Spotfire data functions and MATLAB via TSSS is working as designed, but it might be possible (in a future release) to find ways to redesign it so that it would run faster.

I will post an enhancement request that asks for speed improvements in data transfer between Spotfire data functions and MATLAB via Spotfire Statistics Services (TSSS).

Answer 1

[Michel]

It has be a long road with a lot of work, so I thought I would share our experience with the Tibco Community.

At this stage after months of trying we have not been able in our current IT infrastructure to use Tibco's Spotfire with Data Functions in Mathwork's Matlab with the expected performance. Our application also calls on DBMSs but that works fine.

The symptoms are as follows:

- the processing time by Spotfire for the analysis we are performing takes overall 36 min on the Server.

- the Matlab code in the Data Function starts processing only after an 11 min delay.

- The Matlab in the Data Function takes 23 min to run on the server. The same Matlab code run outside of Spotfire takes 9 min on a local PC.

The CPU (4), Memory (16Gb) and Network configurations between the server and the local PC installations are similar and never show over 40% utilization.

We have had numerous conversations with Tibco Support (case 101257893), Mathworks (Vendor of Matlab; case [ ref:_00Di0Ha1u._50031hTckR:ref ]) and IBM (administrators of our servers & network).

Future "brave" souls might try the following:

1) Compare performance of Spotfire & Matlab on server called by client on local PC (current set up) with Spotfire & Matlab on local PC compared with Spotfire & Matlab on server called by client on server.

2) Move installation from virtual server to stand-alone server.

3) Optmize Matlab code for Spotfire: transfer data to the Data Function as much as possible as rows rather than columns; use Julian rather than standard dates. We also put in Tibco Ideas suggestions for: functionality to pass data as Matlab Table objects rather than Structure objects; automatic translation of dates format from Spotfire to Matlab

Should you have any luck with the issue in the near future, please let us know. It might still be time for us to revisit.