The president joked that some of the audience members had trickled out before his speech at the Clinton Global Initiative