This part is expected. The same filters have to apply on both sides, because otherwise if a file was excluded on the source but present on the dest, rclone would think it is supposed to be deleted during the sync.
I'm pretty sure that what's happening here is that rclone is trying to rmdir the directory, but this is silently failing because the directory isn't actually empty. It appears empty because file.log is excluded with filters, but the local backend knows that file.log is actually there, and therefore refuses to remove the directory. (If you add -vv I think you should see a debug log confirming this.)
This is an issue with no perfect solution at the moment, although I actually submitted a PR recently that proposes a fix, so hopefully there will be a better solution soon:
In the meantime, what I'd recommend is either to delete the file (remove filter or manually delete), move it out of your tree, or keep it as is but accept that you won't be able to delete that directory. (I run into this myself constantly with .DS_Store files on macOS -- and I've generally found that trying to exclude them with filters causes more problems than it solves. That should change once the PR is merged.)
If you're feeling adventurous, there's also the --delete-excluded flag... but it can really be quite destructive so I would urge you to make sure you understand how it works before trying it. And --dry-run first! (In most cases, I think you are better off just manually deleting the problematic file.)