Your original .gitignore should work, tho
venv/ makes it specifically about a directory. git rm -rf -cached venv is not correct, you need two dashes --cached. If that's what you ran, you never actually removed the venv files from Git. Check git log --stat to see if the files were removed.