"Generative AI Only Knows What It Knows"