If it is rabbit season and your AI says it is duck season, that is bad AI.

If your dog points at a duck you can shoot, that’s good AI.

If your dog points at a bald eagle and you don’t double-check, that’s bad AI, and your fault for not double-checking.

Giving your dog your shotgun so it can shoot the duck is bad AI.

If your dog sees a duck flying behind your brother and shoots at it, hitting your brother in the face, that is bad AI, but it is your fault for not including “don’t shoot people” in your prompt. If it’s not prohibited specifically in the prompt, LLM agents (being incapable of context) can and will do it in order to accomplish the goal you set in your prompt.

If your prompt includes “don’t shoot non-duck animals without asking me first,” you made a beginner’s error. All the agent has to do is inform you (whether you are looking or not). Since the prompt does not say “without asking me first and waiting for my answer,” you get what your prompted.