How to Fix a Stuck Cisco Firepower Deployment Notification & How to Delete a Stuck Deployment
Did you ever deploy on your Cisco Firepower/FTD environment and then see something like this?
Let’s start by just deleting a stuck deployment notification, and then I’ll show you how to clear a process on a deployment issue
Deleting a Stuck Deployment Notification
To find the deployment notification that you want to delete, first we’ll log into the FMC and move to root.:
expert
sudo su –
Now we use the OmniQuery.pl tool to query the database and grep for “\ 7\ ” to find the running tasks. Find the uuid of the task that is running, for simplicity I just have the one task listed.To find the notification number:
#OmniQuery.pl -db mdb -e “select status,category,hex(uuid),body from notification;” | grep ” \ 7\ *
Then we make a query that deletes that notification, just like so
To delete the deployment notification:
OmniQuery.pl -db mdb -e ‘update notification set status=13 where status=7;’
or:
#OmniQuery.pl -db mdb -e “delete from notification where uuid-unhex(* (enter notification number here from last command output) *);*
Now if we check again the task is gone!
#OmniQuery.pl -db mdb -e “select status,category,hex(uuid),body from notification;” | grep ” \ 7\ *
Deleting a Stuck Deployment
To find the deployment process that you want to clear, first we’ll log into the FMC and move to root.:
expert
sudo su –
root@firesightmanagement:/var/log# OmniQuery.pl -db mdb -e “select * from notification_status;”
+——–+———–+——-+
| status | label | level |
+——–+———–+——-+
| 1 | info | 1 |
| 2 | success | 1 |
| 3 | normal | 1 |
| 4 | recovered | 1 |
| 5 | disabled | 1 |
| 6 | waiting | 1 |
| 7 | running | 1 |
| 8 | retrying | 1 |
| 9 | suspended | 1 |
| 10 | stopped | 1 |
| 11 | warning | 2 |
| 12 | critical | 3 |
| 13 | failure | 3 |
| 14 | error | 3 |
+——–+———–+——-+
14 rows in set (0.00045 seconds)
root@firesightmanagement:/var/log# OmniQuery.pl -db mdb -e “select status,category,hex(uuid),body from notification where status=7;”
+——–+———————+———————————-+——————————————————————————————————————————————-
| status | category | hex(uuid) | body |
+——–+———————+———————————-+——————————————————————————————————————————————-
| 7 | deployment:category | 4D595DC11057366AA951161CB9A7BD4B | {“property”:”deployment:deployment_initiated_for_the_device”,”arguments”:{“PHASE”:”Phase-0″,”DEVICE_ID”:”339302418939″,”JOB_ID”:”562640821028″},”literal”:null} |
+——–+———————+———————————-+——————————————————————————————————————————————-
1 row in set (0.000325 seconds)
root@firesightmanagement:/var/log# OmniQuery.pl -db mdb -e ‘delete from notification where uuid=unhex(“4D595DC11057366AA951161CB9A7BD4B”);’);’
root@firesightmanagement:/var/log# OmniQuery.pl -db mdb -e “select status,category,hex(uuid),body from notification where status=7;”
root@firesightmanagement:/var/log# OmniQuery.pl -e “SELECT j.JOB_NAME,CURRENT_STATE,DEVICE_ID,d.DEVICE_STATUS, CREATE_TIME, LAST_UPDATE_TIME, LAST_ACTION_TIME from jobs_table j,jobs_device_status d where j.job_id=d.job_id AND CURRENT_STATE=’Deploying’;”db mdb -e “select status,category,hex(uuid),body from notification where status=7;”
root@firesightmanagement:/var/log# OmniQuery.pl -db mdb -e “select status,category,hex(uuid),body from notification where status=7;” ”
+——–+———-+———–+——+
| status | category | hex(uuid) | body |
0 rows in set (0.00032 seconds)
I think this deletes the ‘notification’ but the process still runs in the background, this is true at least for domain management process
I do the following in 6.5
sudo OmniQuery.pl -db mdb -e “select status,category,hex(uuid),body from notification;” | grep “| 7 |”
and then kill the PID
sudo OmniQuery.pl -db mdb -e ‘delete from notification where uuid=unhex(“pid”);’
Great, thank you!
Maybe I am old school (or just old) but how is this a “fix”?
You look for a “running task” by grepping a DB and then you kill the task/process/pid? Shouldn’t this be said to be killing a running task to get rid of a notification telling you it is still running?
Where is the validation that the task you are being notified about is actually completed before killing the notification? What about prevention of the same issue coming back? Those are fixes.
This doesn’t kill the actual process. It only deletes the task from the Notifications list.
If the process is still running it will reappear in the Notifications list after a few seconds.
You can check in the action_queue to see if a certain task is actually still running:
OmniQuery.pl -db mdb -e “SELECT description,target,create_time,last_state_change,TIMEDIFF (last_state_change,create_time) as Duration ,message,state,hidden FROM action_queue WHERE state=3 order by last_state_change;”
If the task is still shown in the queue, it’s still running and deleting it from the Notifications list won’t help.
If it’s not shown in the queue, it just means that it’s a stuck notification which you can then safely delete.
Pascal, that is correct. I wrote this as I had a customer that had a suck process for a year on their firepower
the codes I listed here, clear that out for them
This isn’t meant to stop an active deploy or update, just a stuck notification that is annoying as hell
Hi Pascal,
as you suggested, i can see rows captured with your select query “OmniQuery.pl -db mdb -e “SELECT description,target,create_time,last_state_change,TIMEDIFF (last_state_change,create_time) as Duration ,message,state,hidden FROM action_queue WHERE state=3 order by last_state_change;”
so whats next , how we can clear/delete this from here.
Standalone FTD [well using CDO], had an issue with deployment today, it was stuck, I couldn’t neither discard nor deploy new configs.
There’s a bug: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvr45752?rfs=iqvred
If another device in HA is in a failed state, the deployment will be stuck
To resolve this – break the HA, all interfaces will be disabled except for Management.
I think you’d have to break HA. Did it work after typing in the commands?
I am having a similar issue but rather than deploying it is stating that the registration is in progress and has been like that for a week, even after reboots. I currently cannot get support due to a distributor error on the coverage that we are working to resolve. I have tried to use the show process but there is nothing in the notification table and the Action_queue table shows historical tasks. I cant seem to locate the table showing the registration in progress. Would you happen to know how to find/clear this?
so go into the Devices page and remove the device, if it is showing there at all. If it’s not there, or you have removed it, then go to the CLI of the device and type >configure manager delete. Then add the manager back in with the configure manager add IP_Address password
so you need to reboot the FMC and the FTD, and then register the FTD. I had this issue last week as well, and it takes a while to get this going. Does the device show in Device Manager? If so, you can delete it from the FMC and then re-register it
Dear Todd, Pascal,
Is everyone on this thread having a complete disregard for typos in the commands?
You are using the wrong ” for the Firepower FMC cli.
Other than this the below command has broken syntax – you are closing );’);’ twice where there is only one parantheses open and no need for a second apostrophe.
root@firesightmanagement:/var/log# OmniQuery.pl -db mdb -e ‘delete from notification where uuid=unhex(“4D595DC11057366AA951161CB9A7BD4B”);’);’
People are going to probably use this on production equipment. I do understand that there is a practice of inserting typos into online code samples just to throw off the “Fake programmers” but this is going to hurt the community in the end.
Best Regards,
C
Hi
I have exactly the same problem, only after deleting the message, the deployment in the GUI continues, but when I re-check for the presence of the message, it is missing.
Tried to reboot both FMC and FTD. Nothing helps.
what code version are you running?
CLI procedure to upgrade FMC?