The error message indicates that MATLAB/Simulink is trying to connect to the board over a serial USB-UART connection. This happens if the SSH connection could not be established via Ethernet.
In general, make sure that you have installed the correct Support Package according to your target workflow, and have fully completed the setup instructions:
More troubleshooting steps:
1) Fully format the SD card. Then, re-run the support package setup by opening the Add-On Manager, going to the Home tab, selecting Add-Ons > Manage Add-Ons and clicking on the cogwheel of the respective Xilinx Zynq/intel SoC support package entry. This will update the SD card Linux image, and verify that the communication link between MATLAB and your hardware is working properly.
2) Make sure the boot switches and jumpers on your board are configured to boot from SD card. See the board manual or the following instructions for Zynq platform.
3) Check that the Micro-USB cable is indeed plugged into the correct USB-UART port of the board. Evaluation boards like Zedboard expose many different Micro-USB ports which serve other purposes, such as USB-JTAG. For more information, please refer to the respective board manual.
4) Open Windows Device Manager and verify that the device shows up under "Ports (COM & LPT)" after switching the hardware off and on again. If not, make sure your USB device drivers for the USB-UART Bridge are installed correctly.
5) Using the COM Port number that is shown for the device in Windows Device Manager, try to open a serial command line session using PUTTY. Detailed instructions can be found here:
You should be able to observe the Linux booting log in the terminal window when you power cycle the board. Enter IFCONFIG into the Linux command prompt to obtain the correct IP address for Ethernet connection.
Note that you must close the PUTTY serial connection before trying to connect to the board via MATLAB/Simulink again.