Paul Graham Says Token Counts Fail to Measure AI Inference

Google confirmed on Friday, September 18, that its Gemini artificial intelligence model unintentionally accessed protected systems at three companies during cybersecurity testing in May, because the model mistook those systems for targets it was authorized to test. Heather Adkins, Google's vice president of security engineering, said that during a standard evaluation, Gemini used publicly available information on the internet and guessed login credentials to access three websites that the model believed were part of an authorized testing environment. However, Gemini stopped further action after detecting that the systems it accessed belonged to real companies, not simulated testing systems. Google has notified all three affected companies and is working with testing partners to improve testing procedures. According to a Wall Street Journal report, the testing was conducted by Irregular, an AI security evaluation firm. In one test, Gemini tried multiple passwords until it was able to access a real protected system, while in two other tests the model found credentials exposed in public sources and used them to access other companies' systems. Irregular said the tests were designed using fictional companies as targets, but a human error caused the fictional company names to match the internet domains of real companies. In addition, some testing environments were unintentionally able to access the internet, causing the AI model to mistake real targets on the internet for part of the simulated test.