dhtmlxMenu

How can I add a new menu mode.

I’d like to to build a vertical menu with submenus laying under the menuitem

There is no easy way to define custom mode.
You can check _openPanel method ( dhtmlxmenubar.js , line 1420 ) it describes how position of submenu calculated for each mode