Basically deep of serialization can’t be controlled by API, so any changes must be made directly in code
You can look at next code line
dhtmlxtree.js , line 111
for (var i=0; i<itemNode.childsCount; i++)
out+=this._serializeItem(itemNode.childNodes[i]);
this is code called recursive to itterate through all tree, so any kind of check can be added here to skip branches