Start with one job, not a tool. Write down the task, how you'll measure it and what the agent would need access to. Then test two or three options on your own real examples, check what each can see and do and where your data goes, compare the true monthly cost at your volume, and pick the one that does the job with the least access.
Seven steps
- Name one job. Write the task in one sentence and how you'll know it's done well: hours saved, response time, fewer misses.
- List the access it needs. Which systems and which data the job actually uses. Anything beyond that is risk without benefit.
- Shortlist two or three tools. Include the simplest option, even if it isn't called an agent. Sometimes a template or a rule does the job.
- Test on your real examples. The everyday cases and the messy ones. Note wrong answers, speed and what it does when it isn't sure.
- Check the terms. Where your data goes, whether it trains their models, how long it's kept, how to delete it and how to turn the agent off.
- Compare the real cost. Monthly fee plus usage at your volume, plus the time a person spends reviewing its work.
- Start narrow. Launch with limited access and approval on anything that sends, pays or deletes. Widen it once it's earned it.
What to write down for each option
| Question | Why it matters |
|---|---|
| How often was it right on our examples? | Demos use easy cases. Yours won't be. |
| What does it do when it isn't sure? | Asking is safe. Guessing is how wrong emails go out. |
| What access does it need? | Less access, less to go wrong. |
| Can a person approve its actions? | Your safety net for anything that can't be undone. |
| Where does our data go? | Two companies may receive it: the agent vendor and the AI model provider. |
| What does it cost per month at our volume? | Usage fees grow with you. |
| Can we export our settings if we leave? | So you're not locked in. |
Want it done for you? An AI Agent Evaluation tests the options on your work and ends in a written recommendation: use it, fix it first, or skip it.
Quick answers
Should the vendor do the evaluation?
They should help, but the final check should come from someone who isn't selling you the agent. Ask for a written recommendation you can act on with any vendor.
How many tools should I test?
Two or three is plenty. Always include the simplest option, so you know whether an agent is needed at all.