How to debug Ansible

There are a couple of ways to get more information about why a command/playbook is failing. These are listed more or less, in order growing more verbose. This is a pretty long post mostly due to the example output.

Debug module in playbook

The second most common method is using the debug module in your playbook to output content of a variable during a playbook run so you can verify it. This is less desirable as you have to remember to put these in your code, and it can lead to really lengthy playbook output if you have variables with a lot of output. Below is an example of how to use it though.

---
- name: Gather IOS Facts
 hosts: testciscocore.test.com
 gather_facts: false
 connection: network_cli
 vars:
   ansible_network_os: ios

 tasks:
 - name: Gather facts
   ios_facts:
     gather_network_resources: all
   register: ios_result

 - name: debug
   debug:
     msg: "{{ ios_result }}"

This will output the contents of ios_result to stdout on the screen during a playbook run.

Verbose Parameter

The first, and most common way to get more information is using the -v parameter on the CLI. Note: you can add more "v"s to get more verbose output.

Example playbook (gather_facts.yml):

---
- name: Gather IOS Facts
  hosts: testciscocore.biles.com
  gather_facts: false
  connection: network_cli
  vars:
    ansible_network_os: ios

  tasks:
    - name: Gather facts
      ios_facts:
        gather_network_resources: all
      register: ios_result

Output with standard run of ansible-playbook gather_facts.yml

ansible@blueansible:~$ ansible-playbook gather_facts.yml -u ansible -k
SSH password:

PLAY [Gather IOS Facts] ***********************************************************************

TASK [Gather facts] ***********************************************************************
[WARNING]: default value for `gather_subset` will be changed to `min` from `!config` v2.11 onwards
ok: [testciscocore.test.com]

PLAY RECAP ***********************************************************************
testciscocore.test.com   : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Pretty basic output here, but now, if we run the same playbook with -vv we'll see a lot more info.

ansible@blueansible:~$ ansible-playbook gather_facts.yml -u ansible -k -vv
ansible-playbook 2.9.10
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ansible/.local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Apr 16 2020, 01:36:27) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
Using /etc/ansible/ansible.cfg as config file
SSH password:

PLAYBOOK: gather_facts.yml *****************************************************************************************************************************************************************************
1 plays in gather_facts.yml

PLAY [Gather IOS Facts] ********************************************************************************************************************************************************************************
META: ran handlers

TASK [Gather facts] ************************************************************************************************************************************************************************************
task path: /home/ansible/gather_facts.yml:10
[WARNING]: default value for `gather_subset` will be changed to `min` from `!config` v2.11 onwards
ok: [testciscocore.test.com] => changed=false
  ansible_facts:
    ansible_net_all_ipv4_addresses:
    - 192.168.3.2
    - 192.168.7.2
    - 192.168.3.2
    - 192.168.252.2
<lines omited for brevity>
META: ran handlers
META: ran handlers

PLAY RECAP *********************************************************************************************************************************************************************************************
testciscocore.test.com   : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

With -vv you'll see variable contents output without having to use a debug task in the playbook. You will also see that it prints out what Ansible is using for host, and inventory files, as well as python version. This can be very useful when you run into issues and need to make sure Ansible is using the right host files or inventory files. With -vvv and -vvvv you'll see even more output about what's being done in the background.

ansible@blueansible:~$ ansible-playbook gather_facts.yml -u ansible -k -vvvv
ansible-playbook 2.9.10
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ansible/.local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Apr 16 2020, 01:36:27) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
Using /etc/ansible/ansible.cfg as config file
SSH password:
setting up inventory plugins
host_list declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
script declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Parsed /etc/ansible/hosts inventory source with ini plugin
Loading callback plugin yaml of type stdout, v2.0 from /home/ansible/.local/lib/python3.6/site-packages/ansible/plugins/callback/yaml.py

PLAYBOOK: gather_facts.yml *****************************************************************************************************************************************************************************
Positional arguments: gather_facts.yml
verbosity: 4
ask_pass: True
private_key_file: /home/ansible/.ssh/ansible_rsa
remote_user: ansible
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/etc/ansible/hosts',)
forks: 5
1 plays in gather_facts.yml

PLAY [Gather IOS Facts] ********************************************************************************************************************************************************************************
META: ran handlers

TASK [Gather facts] ************************************************************************************************************************************************************************************
task path: /home/ansible/gather_facts.yml:10
<testciscocore.test.com> attempting to start connection
<testciscocore.test.com> using connection plugin network_cli
<testciscocore.test.com> local domain socket does not exist, starting it
<testciscocore.test.com> control socket path is /home/ansible/.ansible/pc/1890761c07
<testciscocore.test.com> local domain socket listeners started successfully
<testciscocore.test.com> loaded cliconf plugin ios from path /home/ansible/.local/lib/python3.6/site-packages/ansible/plugins/cliconf/ios.py for network_os ios
<testciscocore.test.com>
<testciscocore.test.com> local domain socket path is /home/ansible/.ansible/pc/1890761c07
<testciscocore.test.com> ESTABLISH LOCAL CONNECTION FOR USER: ansible
<testciscocore.test.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/ansible/.ansible/tmp/ansible-local-10573803snxlu6a `"&& mkdir /home/ansible/.ansible/tmp/ansible-local-10573803snxlu6a/ansible-tmp-1593572389.8371599-1057499-117182036127322 && echo ansible-tmp-1593572389.8371599-1057499-117182036127322="` echo /home/ansible/.ansible/tmp/ansible-local-10573803snxlu6a/ansible-tmp-1593572389.8371599-1057499-117182036127322 `" ) && sleep 0'
<testciscocore.test.com> Attempting python interpreter discovery
<testciscocore.test.com> EXEC /bin/sh -c 'echo PLATFORM; uname; echo FOUND; command -v '"'"'/usr/bin/python'"'"'; command -v '"'"'python3.7'"'"'; command -v '"'"'python3.6'"'"'; command -v '"'"'python3.5'"'"'; command -v '"'"'python2.7'"'"'; command -v '"'"'python2.6'"'"'; command -v '"'"'/usr/libexec/platform-python'"'"'; command -v '"'"'/usr/bin/python3'"'"'; command -v '"'"'python'"'"'; echo ENDFOUND && sleep 0'
<testciscocore.test.com> EXEC /bin/sh -c '/usr/bin/python3.6 && sleep 0'
Using module file /home/ansible/.local/lib/python3.6/site-packages/ansible/modules/network/ios/ios_facts.py
<testciscocore.test.com> PUT /home/ansible/.ansible/tmp/ansible-local-10573803snxlu6a/tmpetl415e0 TO /home/ansible/.ansible/tmp/ansible-local-10573803snxlu6a/ansible-tmp-1593572389.8371599-1057499-117182036127322/AnsiballZ_ios_facts.py
<testciscocore.test.com> EXEC /bin/sh -c 'chmod u+x /home/ansible/.ansible/tmp/ansible-local-10573803snxlu6a/ansible-tmp-1593572389.8371599-1057499-117182036127322/ /home/ansible/.ansible/tmp/ansible-local-10573803snxlu6a/ansible-tmp-1593572389.8371599-1057499-117182036127322/AnsiballZ_ios_facts.py && sleep 0'
<testciscocore.test.com> EXEC /bin/sh -c '/usr/libexec/platform-python /home/ansible/.ansible/tmp/ansible-local-10573803snxlu6a/ansible-tmp-1593572389.8371599-1057499-117182036127322/AnsiballZ_ios_facts.py && sleep 0'
<testciscocore.test.com> EXEC /bin/sh -c 'rm -f -r /home/ansible/.ansible/tmp/ansible-local-10573803snxlu6a/ansible-tmp-1593572389.8371599-1057499-117182036127322/ > /dev/null 2>&1 && sleep 0'
[WARNING]: default value for `gather_subset` will be changed to `min` from `!config` v2.11 onwards
ok: [testciscocore.test.com] => changed=false
  ansible_facts:
    ansible_net_all_ipv4_addresses:
    - 192.168.3.2
    - 192.168.7.2
    - 192.168.3.2
    - 192.168.252.2
<lines omitted for brevity>
META: ran handlers
META: ran handlers

PLAY RECAP *********************************************************************************************************************************************************************************************
testciscocore.test.com   : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

DEBUG_ANSIBLE

This method will give EXTREMELY verbose output to the screen, but is sometimes needed when really low level errors are happening. You can place ANSIBLE_DEBUG=1 in front of your CLI command to get the gritty details of what's going on with your playbook. Output for this is much to verbose to include here, but give it a try on your own with this example. It gives you just about every step happening during the run. You can also combine this with the -vvvv parameter to include both sets of debugging information.

DEBUG_ANSIBLE=1 ansible <host> -i <inventory_file> -m setup