There are few steps to setup MATLAB working in Emacs and few more to get working in babel orgmode. This works in MacOS 10.12.6, Emacs 25.3.1 and MATLAB 2017b (also tested in El Capitan with R2016b and Emacs 24.5.1).

For the first step, I found one response from MathWorks Support Team. The basic idea is that the stack size in shell for Mac computers has to be multiple of 4KB, whereas Emacs has does not have a fixed stack size. In the link above there is a script which can launch MATLAB using 4KB stack size.

Setup MATLAB with Emacs

Put the script in a folder of your choice, let's say this is ~/bin forlder in your $HOME directory. Create the file matlab_emacs_wrapper with the following content and make the file executable (eg. chmod 755 matlab_emacs_wrapper) :

#!/bin/bash
# https://ch.mathworks.com/matlabcentral/answers/319655-why-does-matlab-r2016b-crash-when-launched-from-matlab-emacs?
shell_stack_size=$(ulimit -s)

adjusted_stack_size=$[4*((($shell_stack_size-1)/4)+1)]

ulimit -s $adjusted_stack_size && /Applications/MATLAB_R2017b.app/bin/matlab $@

Try to run this script to check if you can launch MATLAB successfully.

Emacs init file configuration for running MATLAB scripts

Make sure you have matlab-mode for Emacs. Add the following in you Emacs init file.

(load-library "matlab-load")
(setq matlab-shell-command "/Applications/MATLAB_R2017b.app/bin/matlab")
(customize-set-variable 'matlab-shell-command "~/bin/matlab_emacs_wrapper")

Now you should be able to run M-x matlab-shell and launch MATLAB from Emacs. If you write a new MATLAB file, let's say test.m, you should be able to evaluate the content of a MATLAB region ¹ using Ctrl + Option + RET.

Setup MATLAB with babel orgmode

Add the following in you Emacs init file

;; setup matlab in babel
(setq org-babel-default-header-args:matlab
  '((:results . "output") (:session . "*MATLAB*")))

;; list of babel languages
(org-babel-do-load-languages
 'org-babel-load-languages
 '((matlab . t)
   ))