Sometimes there is a necessity to show some App page in Modal Dialog.
For example, show some settings on custom action click.
It is simple to show the dialog, but standard approach of closing it doesn't work because the App and SharePoint site can be hosted on different domains. And we can't just commit popup from cross-domain frame document.
To close the dialog you can use HTML5 postMessage API.

If we speak about some custom action, and, to be correct, about custom action that was created declaratively with HostWebDialog property set to true, you can use messages CloseCustomActionDialogRefresh to close the dialog and refresh the page or CloseCustomActionDialogNoRefresh to close but without refresh:

var target = parent.postMessage ? parent : 
    (parent.document.postMessage ? parent.document : undefined); 
if (target) 
    target.postMessage((doRefresh ? 'CloseCustomActionDialogRefresh' : 'CloseCustomActionDialogNoRefresh'), '*');
Server-side:
private void CloseDialog()
{
    this.Context.Response.Write(
        "<script>" + // add type="text/javascript" here. It is removed to avoid syntax highlighter issues
        "    var target = parent.postMessage ? parent :" +
        "        (parent.document.postMessage ? parent.document : undefined);" +
        "    if (target)" +
        "        target.postMessage((doRefresh ? 'CloseCustomActionDialogRefresh' : 'CloseCustomActionDialogNoRefresh'), '*');" +
        "</script>");
    this.Context.Response.Flush();
    this.Context.Response.End();
}
If you open the dialog by custom code (code-created custom action has no HostWebDialog property so it is kind of a dialog opened by custom code) SharePoint doesn't listen to the messages described above. But you can use CloseDialog message and dialogReturnValueCallback to refresh the page:
// open dialog with callback handler example:
SP.UI.ModalDialog.showModalDialog({ 
    url: 'your_app_page_url_or_some_proxy_page_url', 
    // ...
    dialogReturnValueCallback: function CallDETCustomDialog(dialogResult, returnValue) { SP.UI.ModalDialog.RefreshPage(SP.UI.DialogResult.OK); } });

// posting a message
var target = parent.postMessage ? parent : 
    (parent.document.postMessage ? parent.document : undefined); 
if (target) 
    target.postMessage('CloseDialog', '*');
Server-side:
private void CloseDialog()
{
    this.Context.Response.Write(
        "<script>" + // add type="text/javascript" here. It is removed to avoid syntax highlighter issues
        "    var target = parent.postMessage ? parent :" +
        "        (parent.document.postMessage ? parent.document : undefined);" +
        "    if (target)" +
        "        target.postMessage('CloseDialog', '*');" +
        "</script>");
    this.Context.Response.Flush();
    this.Context.Response.End();
}

That's all, have fun!