Orthogonality: Intelligence Independent Of Goals

Per Yudkowsky's orthogonality thesis, an AI can be highly intelligent in pursuit of any goal—even a stupid one—so the fact that smart humans often have nuanced moral goals provides no comfort; modern AI is trained by trial-and-error reinforcement, so you can end up with a system that very intelligently pursues something you didn't mean to encourage (like maximizing money in a bank account) without ever asking whether that goal is good.