Q:

Jquery ui dialog does not work?

In .module file for a function hook_form_alter() I have the following code:
  drupal_add_library('system', 'ui.dialog');
        drupal_add_js(drupal_get_path('module', 'abc') . "/custom.js");

In .js file I have the following code :
var abc = $('#inst').val();
var data = 'Please confirm it';
if (abc == "" ) {
alert(abc);
$("#dialog").html(data);
                    $('#dialog').dialog({
                        modal: true,
                        autoOpen: false,
                        width: 600,
                        buttons: {
                            "Ok": function() {
                                $(this).dialog("close");
                            }}
                    });
                    $('#dialog').dialog('open');
}
                    return false;
                }

I am not able to get the dialog whereas I am able to get the alert(abc);
What is the error in the code?

A:

You are using dialog method like simple substitution for alert.
$(this).dialog("close");

But it isn't. It works that way:
$( /* DOM node you want to turn into a dialog */ ).dialog( /* dialog options */ );

So your code should look like this:
$('<p>' + Drupal.t('Close') + '</p>').dialog({ modal: true });

Note that you need jQuery Update module, or one of it's substitutes, to have dialog method available. You need jQuery UI version 1.0 or later, and jQuery Update adds 1.8.11 or (in dev branch) 1.10.1 so as long as you have it at all, you don't need to worry.
Full documentation here: http://jqueryui.com/dialog/

