She made her name in a drama about sadomasochism – and broke the mould again as a porn director in The Deuce. But it’s her hotly debated new film that might prove to be her most transgressive yet