Q:

HitTest for both Spritekit and Scenekit

I have what seems to be a pretty basic ViewController with a SCNView which has an overlaySKScene.
The issue I have is that I don't want the tap to be detected in the underlying self.scene (gameScene in the example below) if it was first detected in a SpriteKit node in the overlay scene.
With the following, both scenes report that a hit occurred even though the tap happened on the SKOverlay scene node actionButtonNode.
ViewController
import UIKit
import SceneKit

class ViewController: UIViewController {
  override func viewDidLoad() {
    super.viewDidLoad()

    sceneView = self.view as? SCNView
    gameScene = GameScene()

    let view = sceneView
    view!.scene = gameScene
    view!.delegate = gameScene as? SCNSceneRendererDelegate
    view!.overlaySKScene = OverlayScene(size: self.view.frame.size)

    let tapGesture = UITapGestureRecognizer(target: self, action: #selector(ViewController.handleTap(_:)))
    tapGesture.cancelsTouchesInView = false
    view!.addGestureRecognizer(tapGesture)
  }

  @objc func handleTap(_ sender:UITapGestureRecognizer) {
    let projectedOrigin = self.sceneView!.projectPoint(SCNVector3Zero)
    let taplocation = sender.location(in: self.view!)
    let opts = [ SCNHitTestOption.searchMode:1, SCNHitTestOption.ignoreHiddenNodes:0 ]
    let hitList = self.sceneView!.hitTest(taplocation, options: opts)
    if hitList.count > 0 {
      for hit in hitList {
        print("hit:", hit)
      }
    }
  }
}

OverlayScene
import UIKit
import SpriteKit

class OverlayScene: SKScene {
  var actionButtonNode: SKSpriteNode!

  override init(size: CGSize) {
    super.init(size: size)

    self.backgroundColor = UIColor.clear

    self.actionButtonNode = SKSpriteNode()
    self.actionButtonNode.size = CGSize(width: size.width / 2, height: 60)
    self.actionButtonNode.position = CGPoint(x: size.width / 2, y: 80)
    self.actionButtonNode.color = UIColor.blue
    self.addChild(self.actionButtonNode)
  }

  required init?(coder aDecoder: NSCoder) {
    super.init(coder: aDecoder)
  }

  override func touchesEnded(_ touches: Set<UITouch>, with event: UIEvent?) {
    let touch = touches.first
    let location = touch?.location(in: self)
    if self.actionButtonNode.contains(location!) {
      print("actionButtonNode touch")
    }
  }
}

A:

I don’t have a fancy solution with code for you but two possible work arounds:
a. avoid the situation by handling all the touches in the main viewcontroller. I do that with the SceneKit and SpriteKit overlay too, keep the logic out of the overlay scene and use it only for what it is intended (a graphic overlay, not a regular interactive SpriteKit scene).  You can test if the actionbutton, or any other, was hit in the Scenekit viewcontroller. If no button was hit the 3D scene was hit.
b. Add a bool to the main Scenekit view controller, isHandledInOverlay and set it in the SpriteKit scene if the button was touched. In the main Scenekit view controller check the bool in handletap and return and do nothing if true.
Edit based on comments:
I am simply suggesting to move this piece of code:
if self.actionButtonNode.contains(location!) {
  print("actionButtonNode touch")
}

to the handleTap function in ViewController. You will have to make a var that holds the SKOverlayScene, which in turn has the actionButtonNode (and any other buttons) as a property, so in handleTap you end up with something like: 
if self.theOverlay.actionButtonNode.contains(taplocation) {
  print("actionButtonNode touch")
} else if self.theOverlay.action2ButtonNode.contains(taplocation) {
  print("action2ButtonNode touch")
} else if self.theOverlay.action3ButtonNode.contains(taplocation) {
  print("action3ButtonNode touch")
} else {
    //no buttons hit in 2D overlay, let's do 3D hittest:
    let opts = [ SCNHitTestOption.searchMode:1, SCNHitTestOption.ignoreHiddenNodes:0 ]
    let hitList = self.sceneView!.hitTest(taplocation, options: opts)
    if hitList.count > 0 {
        for hit in hitList {
           print("hit:", hit)
        }
    }
}

So first you compare the 2D tap location to the 2D coordinates of the button(s) to see if they got tapped, and if not then you use the 2D tap location to see if an object in the 3D scene was tapped using the hit test.
Edit: was just looking at my own (obj c) code and noticed I flip the Y coordinate in the main viewcontroller to get the coordinates in the SKOverlay:
CGPoint tappedSKLoc = CGPointMake(location.x, self.menuOverlayView.view.frame.size.height-location.y);

You may have to do that with the location you feed to the contains function of the button nodes.

