Fix issue when we try and poll for fd -1 by deleting events by argument.