Q) How to mask a field, for example like this: (NN) NNNN-NNNN ? N stands for number.

You can pick input jquery id selectors, and link it using any lib.
For e.g. http://igorescobar.github.io/jQuery-Mask-Plugin/

Step1: Add JS lib

<script src="//cdnjs.cloudflare.com/ajax/libs/jquery.mask/0.9.0/jquery.mask.min.js"></script>

Step2: Set field integration, usually $col['name'] is ID of html element

$opt["add_options"]["afterShowForm"] = 'function(formid) { jQuery("#amount").mask("000.00"); }';
$opt["edit_options"]["afterShowForm"] = 'function(formid) { jQuery("#amount").mask("000.00"); }';
$g->set_options($opt);

For more options, refer: http://igorescobar.github.io/jQuery-Mask-Plugin/

not working for multiple fields
any clues?

$grid["add_options"]["afterShowForm"] = 'function(formid) { jQuery("#contract_StartDate").mask("00/00/0000"); }';
$grid["edit_options"]["afterShowForm"] = 'function(formid) { jQuery("#contract_StartDate").mask("00/00/0000"); }';

$grid["add_options"]["afterShowForm"] = 'function(formid) { jQuery("#contract_EndDate").mask("00/00/0000"); }';
$grid["edit_options"]["afterShowForm"] = 'function(formid) { jQuery("#contract_EndtDate").mask("00/00/0000"); }';
Added JS lib
applied field integration as noted above.

also tried to add pattern for limiting user input date values
.mask('00/00/0000', {'translation': {0: {pattern: /[0-9*]/}}});

Demo code here: http://pastebin.com/KFTQAbR1

However, if you are formatting date, best would be to use date formatter.

# format as date
$col["formatter"] = "date";

# opts array can have these options: http://api.jqueryui.com/datepicker/
$col["formatoptions"] = array("srcformat"=>'Y-m-d',"newformat"=>'m/d/Y', "opts" => array("changeYear" => false));

yes. does work better with formatter=date.
but,
is there a way to limit user input to digits only without setting up a custom validation function.
need to force input of digits with slash.

You can make input field to readonly.

$col["editoptions"]["readonly"]="readonly";
$col["editoptions"]["style"]="background:white;"

This way, only datepicker selected date will be set.