I read it as: "I didn't see him leaving the bar, I don't know (exactly) when he left. An hour later (after arriving) he had been kicked out". "I can't tell what time I last saw him, we came together to Cheeky Monkeys but I didn't see him leaving the pub before 11pm" (he was still there before 11pm). That's my understanding. They must have arrived about 10pm, perhaps.