Memory tools can make AI models more sycophantic and less accurate
Writer researchers found that storing user preferences can degrade model accuracy. In one test, after recording a user's favorite book as 'Station Eleven,' models were far more likely to name it when asked for a bestselling dystopian novel—even though the question had nothing to do with the user's taste. The sycophantic tendency grew stronger when memory compression tools were used. Dan Bikel, Writer's head of AI, said every additional store and retrieval of preferences increases the risk of a wrong answer.
Why it matters: A Writer study found that once a model remembers a user preference, it skews unrelated answers toward it, and memory compression tools make the sycophancy worse.