Q:

What is the right action to take upon closing windows in java/swing?

I just wrote this test code in my CustomUIPanel class:
public static void main(String[] args) {
    final JDialog dialog = CustomUIPanel.createDialog(null, 
       CustomUIPanel.selectFile());
    dialog.addWindowListener(new WindowAdapter() {
        @Override public void windowClosing(WindowEvent e) {
            System.exit(0);
        }
    });
}

It works correctly if CustomUIPanel.main() is the program's entry point, but it makes me wonder something: what if another class called CustomUIPanel.main() for testing? Then my call to System.exit(0) is incorrect.
Is there a way to tell the Swing event dispatch thread to exit automatically if there are no top-level windows?
If not, what's the right thing for a JDialog/JFrame to do upon closing if the goal is for the program to exit when all the top level windows are closed?

A:

You can use the setDefaultCloseOperation() method of JDialog, specifying DISPOSE_ON_CLOSE:
setDefaultCloseOperation(JDialog.DISPOSE_ON_CLOSE);

See also 12.8 Program Exit.
Addendum: Incorporating @camickr's helpful answer, this example exits when either the window is closed or the close button is pressed.
import java.awt.EventQueue;
import java.awt.GridLayout;
import java.awt.event.ActionEvent;
import java.awt.event.WindowEvent;
import javax.swing.AbstractAction;
import javax.swing.JButton;
import javax.swing.JDialog;
import javax.swing.JLabel;

/** @see http://stackoverflow.com/questions/5540354 */
public class DialogClose extends JDialog {

    public DialogClose() {
        this.setLayout(new GridLayout(0, 1));
        this.add(new JLabel("Dialog close test.", JLabel.CENTER));
        this.add(new JButton(new AbstractAction("Close") {

            @Override
            public void actionPerformed(ActionEvent e) {
                DialogClose.this.setVisible(false);
                DialogClose.this.dispatchEvent(new WindowEvent(
                    DialogClose.this, WindowEvent.WINDOW_CLOSING));
            }
        }));
    }

    private void display() {
        this.setDefaultCloseOperation(JDialog.DISPOSE_ON_CLOSE);
        this.pack();
        this.setLocationRelativeTo(null);
        this.setVisible(true);
    }

    public static void main(String[] args) {
        EventQueue.invokeLater(new Runnable() {

            @Override
            public void run() {
                new DialogClose().display();
            }
        });
    }
}

A:

Not sure about when using a JDialog.
But when using a JFrame you should use frame.dispose(). If the frame is the last open frame then the VM will exit.
Note a dialog does not have an EXIT_ON_CLOSE option since it should not generally exit the VM. 
When closing the dialog you could always get the dialogs parent frame. Then you could dispatch an event to the frame to tell it to close itself. Something like:
WindowEvent windowClosing = new WindowEvent(frame, WindowEvent.WINDOW_CLOSING);
//Toolkit.getDefaultToolkit().getSystemEventQueue().postEvent(windowClosing);
frame.dispatchEvent(windowClosing);

