How to Automatically Upload AEwin .DTA Files to a Web Server Using MATLAB or Python?

Hello everyone,

I am working on automating the upload of Acoustic Emission data collected via AEwin, which stores data in .DTA format. My goal is to:

1. Automatically process the .DTA files generated by AEwin.
2. Extract key data from these .DTA files.
3. Upload this data to a web server (such as ThingSpeak or a custom server) in real-time.

What I Have Tried:

• Python: I’ve tried reading the .DTA file using the MistrasDTA Python library, and encountered the error.
• MATLAB: While I haven’t tried MATLAB yet, I am wondering if it could be a better alternative for extracting data from .DTA files and uploading it to a server.

My Questions:

1. Has anyone successfully processed AEwin .DTA files in MATLAB? If so, what functions or toolboxes are recommended for this task?
2. Is there a way to automate this process fully (i.e., automatically upload the data to a web server whenever a new .DTA file is created)?
3. Any suggestions or examples of how to integrate this workflow between AEwin and a server would be greatly appreciated.

Thank you in advance for any help or suggestions you can offer!

[Christopher Stapels]

Not familiar with the format but does this File Exchange entry help? https://www.mathworks.com/matlabcentral/fileexchange/111795-aewin-acoustic-emission-continuous-waveform

[Huzaifa]

Thank you for sharing the function! I see that AEwinWaveRead.m is designed to handle waveform data saved in .txt format, but my data is currently being saved in .DTA format by AEwin. I’m looking to automate the entire process, from reading the .DTA files to uploading the extracted data to a web server (like ThingSpeak or a custom API).

Is there a way to modify this function to handle .DTA files directly, or would I need to first convert the .DTA files to .txt format? If conversion is needed, I would also like to automate that part to avoid manual intervention.

Any suggestions or advice on how to fully automate the process would be greatly appreciated!

Thanks again for your help.

[Vinod]

Can you please provide more details? What generates these files? How is the generation automated? What happens once the files are uploaded online? What is the end goal - just move storage of the file from local machine to the cloud?

[Huzaifa]

Here are additional details about the process:

File Generation: The .DTA files are generated by AEwin software during Acoustic Emission (AE) testing. These files contain critical AE data like amplitude, hit count, energy, etc. AEwin continuously stores these files in a specific folder on my local machine.

File Accessibility: These .DTA files can only be opened in AEwin or Noesis software. When I load them into AEwin, they display various graphs and data tables, like amplitude vs. time, hit distribution, etc. (I’ve attached an image showing the interface in AEwin).

End Goal: The objective is to automatically process the .DTA files as they are generated, extract key data points (e.g., amplitude, event time, channel), and upload this data to a web server (such as ThingSpeak) for real-time monitoring. This will help in tracking AE activity remotely and allow decision-making based on real-time data.

If MATLAB can automate this workflow, I'd appreciate any suggestions on the best approach or toolboxes to use.

Thanks again for your help!

[Huzaifa]

Right now, AEwin has a Line Display option where the real-time data is shown. Is it possible to capture that data? I mean, the data would display and simultaneously get uploaded to a server in real-time through MATLAB.

Answer 1

[Hitesh]

Hi Huzaifa,

You can automate the whole process in MATLAB in three phases:

• Detection of new .DTA file
• Reading and processing .DTA Files
• Uploading Data to a Web Server

Detection of new .DTA files:

• Directory Monitoring: Use MATLAB's dir function to periodically check for new files in the directory where AEwin saves the .DTA files.
• Timer Objects: Use MATLAB's timer objects to run the directory check at regular intervals.
• For more information on "how to programmatically detect a change in a directory". Please refer to below MATLAB answer: https://www.mathworks.com/matlabcentral/answers/99277-how-do-i-programmatically-detect-a-change-in-a-directory

t = timer('TimerFcn', @checkForNewFiles, 'Period', 60, 'ExecutionMode', 'fixedRate');
start(t);

function checkForNewFiles(~, ~)
    files = dir('path_to_DTA_files/*.DTA');
    % Read and process file
end

Reading and processing .DTA Files:

• We can read the .dta file by copying into .txt file so that we can convert the .dta file into format readable by readtable.

filename = 'file_name.DTA'; 
% copy the file to a legible format 
filename2 = [filename(1:end-1), 'txt']; 
copyfile(filename, filename2);

% read table of legible file type
T = readtable(filename2, 'ReadVariableNames', 0);

% delete the file that we created bc we don't need it anymore
delete(filename2);

Uploading Data to a Web Server:

• Setup the URL and Request Options:URL defines the server endpoint where the data will be uploaded.Request options determines the format in which the data is being sent.
• Send Data to the Server:The "webwrite" function is used to send the jsonData to the specified URL with the defined options.
• For more information on "webwrite" and "weboptions". Pease refer to below MATLAB documentation:
• Webwrite : https://www.mathworks.com/help/matlab/ref/webwrite.html
• WebOptions : https://www.mathworks.com/help/matlab/ref/weboptions.html

url = 'http://yourserver.com/api/upload';
options = weboptions('RequestMethod', 'post', 'MediaType', 'application/json');
response = webwrite(url, jsonData, options);

[Christopher Stapels]

Since the OP intent is to use ThingSpeak, you might replace the function checkForNewFiles with a MATLAB analysis script and use the TimeControl app to replace the timer object.

[Huzaifa]

Thanks for your answer. I tried this one and even used various Python libraries to access the .dta file, but I keep encountering the same error that access is not allowed. I also contacted AEwin's customer service, and they mentioned that they cannot grant access due to data security reasons.

MATLAB Error:

Error using copyfile

Access is denied.

Now, AEwin has a line display option that shows real-time data. Now, how can this data be captured and uploaded to a server? Does anyone have any idea?