Q:

When removing a file it still is present on the machine

When deleting a node the files attached to that node are removed from the server just fine. However if I edit the node and remove the file there, it stays on the server even after deleting the node.
Is this normal? Is there a way to fix that?

A:

Permanent files have a status FILE_STATUS_PERMANENT.

Indicates that the file is permanent and should not be deleted.
Temporary files older than DRUPAL_MAXIMUM_TEMP_FILE_AGE will be
  removed during cron runs, but permanent files will not be removed
  during the file garbage collection process.

When you remove a file from a node, the file_managed record may simply be marked as status = 0 or anything other than FILE_PERMANENT, in which case, it's deleted during system_cron (if no entry in the file_usage table).
Basically, if you have cron running, it should be fine.

