Git will not delete untracked files, and that's a good thing. For example, it's very common to ignore artifacts of your build process like compiled files. You don't want Git deleting those automatically.
You can run git clean -d -f -x to delete all untracked and ignored files.
You could write a Git post-checkout hook to run git clean every time you checkout or switch branches, but client-side hooks are not checked in with the project; only you will be able to use it. It doesn't scale if you have other contributors. And it will happen every time you checkout, you might think you want that until Git goes and deletes something important you threw into the project temp directory. And it isn't really Git's purpose to keep your project directory tidy. Git tracks content.
Most projects solve this using their build tool to write a "clean" command which would run git clean and any other tasks to clean up the project. Then you can either run it manually as needed, or as part of an automated build process. Which tool would depend on your programming language. Common tools include make or rake (Ruby).
Here's a simple example of a Rake target run with rake clean.
task :clean do sh 'git clean -d -f -x'end