The speaker was wasting significant money on API tokens because his disorganized file structure forced AI to build agents for context retrieval, when organized files would allow direct access to needed information.