zabbix_sender failed items debug

Once upon a time, you need to send a value to Zabbix from a script. Zabbix_sender comes to the rescue: zabbix_sender -z "server IP address" -p 10051 -s "host in zabbix" -k "item key" -o "item value"

But, what happens if for some reason Zabbix won’t accept the value (say, wrong item type?). It’ll just fail (non-zero), without any error message. Well, let’s try to debug (notice the added -vv):

zabbix_sender -vv -z "server IP address" -p 10051 -s "host in zabbix" -k "item key" -o "item value"
Info from server: "Processed 1 Failed 1 Total 1 Seconds spent 0.003079"
sent: 1; skipped: 0; total: 1

Oh wow, that’s helpful. I already know it’s failed. I want to know why.

One would assume that if this information isn’t available at zabbix_sender directly, it must be in Zabbix server logs (/var/log/zabbix/zabbix_server.log ot something like that). One would be wrong.

Then, one would think that increasing DebugLevel to maximum in zabbix_server.conf and restarting Zabbix server would surely shed some light on the issue. After digging through megabytes of irrelevant information, it would be clear that one is wrong again. In summary, there’s no information available as to why a particular item fails to accept a value. Zabbix forces you to scry.

Oh, but wait, let’s make this even more interesting. Say, you need to send multiple items. Fortunately, zabbix_sender can accept properly formatted files ($host $item $value) on input:

zabbix_sender -vv -z "server IP address" -p 10051 -i zabbix.input
Info from server: "Processed 245 Failed 5 Total 250 Seconds spent 0.003079"
sent: 250; skipped: 0; total: 250

Marvellous, isn’t it? WHICH ONES? At this point, one wouldn’t hope to find this information in the logs. And that’s correct, Zabbix doesn’t log it. The solution is to send values one by one and see zabbix_sender responds. Not too difficult, and can even be automated, but why oh why it can’t be printed to stderr or somewhere in the logs? Good, transparent, clear logging is an essential component of any system that aims for “enterprise”. Sorry, but Zabbix isn’t there yet.

Just so it wouldn’t be a completely whining post, I’ll list some common issues to check:

1. hostname in zabbix_agentd.conf and hotstname in Zabbix database are different.
2. Typo in item key.
3. Item value/type mismatch.