The "plain" master is meant to be used with sites that just have a single "cover page" built from layout includelets, or provide their own navigation. No navigation between pages is provided.
The "tabbed" master, selected by default, provides a row of navigation tabs integrated with the navigation tabs generated by the openacs default master template. Each page is represented by a tab, and the page's title provides the label for the tab. As with the default openacs theme, other tabs ("Member", "Administration", etc) and subtabs for the "Administration" page (if enabled) are configured by modifying subsite parameters in the "Navigation Tabs" section.
The third option, which leaves the subsite master untouched, should only be used if you have implemented your own custom theme and master template that integrates navigation tools with layout manager pages. The default openacs master template doesn't work well with the Layout Manager.