Troubleshooting

Most important log files

When have faced an issue on your Zentyal server deployment and you need to start with the troubleshooting [1] process, it is extremely important that you are familiar with the log files. Thease can provide you important clues that will help you to debug the issues and correct them.

[1]Troubleshooting: https://en.wikipedia.org/wiki/Troubleshooting

Below you have a list of the most important log files of Zentyal:

SSH access attemps:

Do you suspect that there might have been a security breach? Do you need to know about the user logins and usage of the sudo command? Check out the SSH access attempts:

/var/log/auth.log
Zentyal modules status:

Are there errors in the services managed by Zentyal? Check out the Zentyal module status:

/var/log/zentyal/zentyal.log
System and daemons logs:

Not quite sure what is going wrong and where? Start by checking out the system and daemons logs:

/var/log/syslog
Installed software:

Facing issues when installing modules or did some services stop running? Check out the installed software:

/var/log/zentyal/software.log
/var/log/dpkg.log
Domain Controller logs:

Are your users having issues when authenticating to the domain? Or are you facing issues when adding Zentyal as an additional DC? Check out the Domain Controller logs:

/var/log/samba/samba.log
Mail module logs:

Hitting issues with the email services (delivery, spam, sent/received messages)? Check out these Mail module logs:

/var/log/mail.log
/var/log/mail.err
SOGo logs:

If your issues are related with webmail or groupware, check out the SOGo logs:

/var/log/sogo/sogo.log
OpenVPN module connections logs:

Issues with your VPN connections? Check out the OpenVPN module connections logs:

/var/log/openvpn/
Proxy module logs:

If you are facing issues with the proxy service, you need to review the Proxy module logs:

/var/log/squid/
Jabber module logs:

If your issues are related with the IM module, check out the Jabber module logs:

/var/log/ejabberd/

Most important commands

Besides being familiar with the logs that store the executed server and service events, it is important to know the commands that allow you to debug the system. Below you can find a list of the most common and useful commands:

Broken packages:

dpkg -l |egrep -v '^ii|rc' | awk '{if(NR>5)print}'

Network outages:

grep 'Link is Down' /var/log/syslog

Errors in the internal Samba database:

samba-tool dbcheck --cross-ncs

Zentyal module status and status of the daemons:

zs dns status
systemctl status bind9
journalctl -u bind9

MySQL databases status:

mysqlcheck -u root -p$(cat /var/lib/zentyal/conf/zentyal-mysql.passwd) \
   --all-databases

Blocked processes:

lslocks

GPOs applied and not applied to a Windows client belonging to the domain:

gpresult

Report of the GPOs applied and denied for a given client:

Through the Microsoft RSAT, in the 'Group Policy Results' menu.

Finally, it is important to know where to look for additional information in case of failures. The following links are especially useful in relation to the Domain Controller module:

  • How to verify the directory replication statuses [2]
  • How to Determine a DCs objectGUID[#]_
[2]https://wiki.samba.org/index.php/Verifying_the_Directory_Replication_Statuses
[3]https://wiki.samba.org/index.php/Verifying_and_Creating_a_DC_DNS_Record

Other useful commands

Like any computer system, Zentyal must be checked daily to ensure its stability. If possible, keep your server monitored by Zabbix [3] or Nagios, etc. These systems will notify you when your Zentyal server is experiencing any incident or anomaly (among other features).

It is always necessary to be familiar with a series of basic commands to obtain information from the server as well as for server maintenance. Below you can find a list of the most useful commands:

Disk status:

These commands provide information about the block devices (hard disks, SSD, USB...) and the amount of disk space available on filesystems:

lsblk
df -hT
CPU workload:

These commands allow you to see the processes running on the system and understand the CPU workloads they are causing:

htop
top -bn1 | egrep -o "load average:.*" | cut -d ":" -f2 | tr -s "," "."
RAM and SWAP use:

This command displays the total amount of free and used physical and SWAP memory in the system, as well as the buffers used by the Kernel:

free -m
Network interfaces:

These commands help to verify and troubleshoot issues related to IP addresses and routes, Internet connectivity, DNS or network connectivity:

ip -4 a
ip route show
ping google.es
dig google.es
dig 8.8.8.8
traceroute google.es
dig @localhost zentyal-domain.lan
Available updates:

These commands provide information regarding the available updates:

apt-get update && apt list --upgradable
Scheduled tasks executed:

This command displays cron related entries, allowing you to check if scheduled tasks have been executed, for example:

grep 'cron' /var/log/syslog
Rejected e-mails:

This command displays the rejected emails, helping you to spot and fix potential email related issues:

egrep 'status=(reject|bounced)' /var/log/mail.log
[4]Zabbix configuration: https://wiki.zentyal.org/wiki/Zentyal_server_monitoring_with_Zabbix