I'd assume the auth handshake would prevent this.

- client connects to SSH server at IP X.X.X.X or hostname SomeHost

- redirected to oAuth server

- Client signs in and receives token scoped to X.X.X.X or hostname SomeHost

- Client provides token to SSH server