how to download data from website?

114 views (last 30 days)
Lilya
Lilya on 23 Mar 2019
Answered: Hitesh on 3 Oct 2023
Dear all,
I am trying to download data from the following website
https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/
my problem is I can not get the files, meaning only the html been located to my computer. (below what I used to locat the link into my machine)
url='https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/'
filename='A2019.nc'
outfilename=websave(filename,url)
what I need is getting the files separtly and read them.
Thanks for the help.
  1 Comment
Juan
Juan on 27 May 2023
To download data from a website, follow these steps:
Identify the data you want to download on the website.
Right-click on the data or the download link.
Select the "Save link as" or "Save target as" option from the context menu.
Choose the destination folder on your computer where you want to save the downloaded data.
Click "Save" to initiate the download.
Wait for the download to complete. The time taken will depend on the size of the data and your internet connection speed.
Once the download is finished, you can access the downloaded data from the specified destination folder on your computer.
It's important to note that downloading data from websites should be done in compliance with applicable laws, website terms of service, and copyright restrictions. Ensure that you have the necessary rights and permissions to download and use the data obtained from websites.

Sign in to comment.

Accepted Answer

Akira Agata
Akira Agata on 26 Mar 2019
How about the following?
url = 'https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/';
str = webread(url);
links = regexp(str,'https://([\w-]+\.)+[\w-]+(/[\w- ./?%&=]*)?(\.nc)','match')';
data = cell(size(links));
for kk = 1:numel(links)
data{kk} = webread(links{kk});
end
By running this, all the .nc files are downloaded and stored in the cell array data.
  12 Comments
Belinda Finlay
Belinda Finlay on 12 May 2020
Thank you for your assistance Akira-san.
Soo Mee Kim
Soo Mee Kim on 11 Jun 2020
Hi, I have the same problem to to download nc files website.
When I tried the following code, both codes gave html file.
Please let me know how to download nc file from the website.
[my own]
fname = 'A2019152.L3m_DAY_PAR_par_4km.nc';
downloadURL = 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2019152.L3m_DAY_PAR_par_4km.nc';
options = weboptions('Username', '<username>', 'Password', '<password>');
websave(fname, downloadURL, options);
[From Akira Agata's answer]
downloadURL = 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2019152.L3m_DAY_PAR_par_4km.nc';
options = weboptions('Username', '<username>', 'Password', '<password>');
data = webread(downloadURL, options);

Sign in to comment.

More Answers (3)

Tracy
Tracy on 1 Aug 2023
It looks like you are using MATLAB's websave function to download the file from the URL. However, the websave function is primarily used to download a file and save it to a specified location. It seems like you are trying to download multiple files, and it won't work as you expect because you are providing a URL of a directory, not a direct link to a specific file.
To download multiple files from the website, you will need to loop through the links in the directory and download each file individually. Additionally, you can use Python with the requests library to achieve this task more easily. Below is a Python script that demonstrates how to download multiple files from the website using the requests library:
python
import requests
import os
url = 'https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/'
output_directory = './downloaded_files/'
# Create the output directory if it doesn't exist
os.makedirs(output_directory, exist_ok=True)
response = requests.get(url)
if response.status_code == 200:
# Parse the HTML content to find links to files
file_links = []
lines = response.text.split('\n')
for line in lines:
if '<a href="' in line:
start_index = line.find('<a href="') + len('<a href="')
end_index = line.find('">', start_index)
file_link = line[start_index:end_index]
if file_link.endswith('.nc'): # Only consider links that point to .nc files
file_links.append(file_link)
# Download each file and save it to the output directory
for file_link in file_links:
file_url = url + file_link
out_file_path = os.path.join(output_directory, file_link)
response = requests.get(file_url)
if response.status_code == 200:
with open(out_file_path, 'wb') as f:
f.write(response.content)
print(f"Downloaded: {file_link}")
else:
print(f"Failed to download: {file_link}")
else:
print("Failed to fetch the URL.")
print("All files downloaded successfully.")

Loria Smith
Loria Smith on 5 Sep 2023
There are several ways of manual web scraping.
  1. Code a web scraper with Python. It is possible to quickly build software with any general-purpose programming language like Java, JavaScript, PHP, C, C#, and so on. ...
  2. Use a data service. ...
  3. Use Excel for data extraction. ...
  4. Web scraping tools

Hitesh
Hitesh on 3 Oct 2023
import requests
# Define the URL of the file you want to download
url = 'https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/A2019.nc'
# Specify the local filename where you want to save the downloaded file
filename = 'A2019.nc'
# Send an HTTP GET request to the URL
response = requests.get(url)
# Check if the request was successful (status code 200)
if response.status_code == 200:
# Open a local file with write-binary ('wb') mode and save the content of the response
with open(filename, 'wb') as file:
file.write(response.content)
print(f"File '{filename}' downloaded successfully.")
else:
print(f"Failed to download file. Status code: {response.status_code}")
There are various methods for manually collecting data from websites:
  1. Coding a Web Scraper with Python: You can create a web scraper using Python, a versatile programming language. Python offers libraries like BeautifulSoup and Scrapy that make web scraping easier.
  2. Utilizing a Data Service: Another option is to use a data service or API provided by the website, if available. This method allows you to access structured data without the need for web scraping.
  3. Leveraging Excel for Data Extraction: Microsoft Excel can also be used for data extraction. You can import data from web pages into Excel and then manipulate and analyze it as needed.
  4. Web Scraping Tools: There are various web scraping tools and software applications designed specifically for data extraction from websites. These tools often provide a user-friendly interface for collecting data without extensive coding.

Categories

Find more on Downloads in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!