Fluctuations of polymer partition functions and collision local times