Azure Timer Function Not Firing

Category: azure functions

Question

BenHow on Tue, 05 Dec 2017 08:41:42


This question is also posted at [https://github.com/Azure/azure-webjobs-sdk-script/issues/2181]

My function did not fire at 09:10am on either Saturday or Sunday as expected.

  • Function App version (1.0 or 2.0-beta): 1.0.11388.0
  • Function App name: asinttest-FA-CostMgmt-dev381
  • Function name(s) (as appropriate): GetConsumptionDataTimerTrigger
  • Invocation ID: 8d9e7e89-9914-4f1f-bdae-7f29d59d109f (last successful, when logged in to portal)
  • Region: North Europe

Repro:

  1. Log in to portal just before function is due to fire
  2. Witness function fire
  3. Go home for the weekend
  4. Log in on Sunday evening to find it has not fired, as confirmed by no entry in my database, nothing in the invocation logs.
  5. Change cron schedule to fire 5 minutes hence, and the function will fire.

Expected:
I expected the function to fire at 09:10 UTC both today (Sunday) and yesterday.

Actual:
The function had not fired.

Related:

  • I'm using a consumption plan, which I'd expected to do the job perfectly, but obviously it is no good if the function will only fire if the application has to be warmed up first.
  • I have not set WEBSITE_TIME_ZONE
  • The cron schedule was set to 0 10 9 * * * until I changed it to 0 16 21 * * * to test it.
function.json:
{
  "generatedBy": "Microsoft.NET.Sdk.Functions.Generator-1.0.6",
  "configurationSource": "attributes",
  "bindings": [
    {
      "type": "timerTrigger",
      "schedule": "0 10 9 * * *",
      "useMonitor": true,
      "runOnStartup": false,
      "name": "myTimer"
    }
  ],
  "disabled": false,
  "scriptFile": "../bin/AzureFunctions.dll",
  "entryPoint": "AzureFunctions.GetConsumptionDataTimerTrigger.Run"
}



Replies

Connor McMahon on Tue, 05 Dec 2017 19:46:49


We have identified this as a problem, and are looking into it. We will let you know when it is resolved.

BenHow on Wed, 06 Dec 2017 10:59:39


Thanks Connor.  Can you provide an ETA?

Mikael Rydström on Thu, 07 Dec 2017 13:21:55


I'd like to add that this is a huge issue for us. We have Azure Functions that trigger on Service Bus (data ingestion) messages, and they just intermittently stop working - and they never self-heal. So the entire data pipeline shuts down.

Since the Service Bus trigger runs on the level of sub 100ms, this generates a lot of calls to the Key Vault for storage credentials, which breaks down with socket exceptions. For some reason, once the Function has died, all I need to do to manually get it running again is:

  • Navigate to my Resource Group, listing the Resources.
  • Click on the Azure Function resource.

That's it. I don't even need to actively stop/start it.

Connor McMahon on Thu, 07 Dec 2017 19:23:58


We are still investigating the issue, so it is difficult to give an ETA. We do have a temporary workaround though.

Create a new app setting named WEBSITE_USE_PLACEHOLDER and set its value to 0. Note that this will turn off our cold start optimizations, so if your function has not been run in a while (and since your function is only ran once a day, this is likely), it will take longer to boot up. 

This is something that you will want to remove as soon as we deploy a fix to this bug. 

Connor McMahon on Thu, 07 Dec 2017 19:26:16


Mikael,

You as well can use the temporary workaround I specified for Ben. Again, you will want to turn this off once the fix is deployed, as you can expect 20+ second start times if you function has not been ran recently with this app setting in place.

Ross_Hammer on Fri, 15 Dec 2017 02:05:52


Connor,

I am experiencing the same issue as reported by BenHow. I tried the workaround fix with the WEBSITE_USE_PLACEHOLDER app setting by manually adding it directly via the Azure Portal, but the precompiled C# Azure Function is still not reliably triggering on the designated schedule. Generally, if I manually run it, it will then timertrigger once as designed and then goes dormant again. Are there any particular tips regarding forcing re-loading the app settings, needing to re-deploy the function with the new app setting, etc? Thanks!

Connor McMahon on Fri, 15 Dec 2017 18:25:03


Ross_Hammer, if you could share your function application name, and the timeframe that you were experiencing these issues, that would be helpful in us identifying if the cause is the same for this issue. If you don't feel comfortable publicly sharing this information, you can share it privately.

Ross_Hammer on Mon, 18 Dec 2017 18:37:21


Thanks, Connor. On Friday, I deleted all of my function resources and redeployed cleanly with the suggested workaround setting, but it only ran once then, and just ran again now that I observed it when I went to check on it:
2017-12-18T18:27:59.810, 269bdb8d-fc07-478d-8e55-983e9ee88c54, "West US"

Ross_Hammer on Thu, 21 Dec 2017 21:48:58


I tried to add a couple follow-up screenshots to my previous post, but I got an error message:
"Body test cannot contain images or links until we are able to verify your account."

After searching through some 5 - 8 pages trying to find out how to get my account verified, and having the support live chat crash on me, I gave up. Since I couldn't find any other way to express my problem, I'm commenting here.

Chris_______ on Tue, 02 Jan 2018 21:18:05


I also seem to be running in to this issue.  I tried the workaround but that doesn't seem to be working for me.  If I disable and then enable my timer function the timer will fire on time about 3 times before it stops firing again.  I have my timer set for 10mins in compiled code which I deployed from visual studio.

2018-01-02T21:16:33.595   43bb5f61-d50e-43ce-b2d4-5d820c88db36   West Central US



Ross_Hammer on Thu, 01 Feb 2018 02:11:26


Been a month and a half since my last check here, compiled Azure Function still doesn't trigger on schedule (doesn't trigger at all, have to manually run it on demand). This worked great when it was a CSX script written in VS 2015, but when we migrated to VS 2017 and a compiled CS library, it stopped triggering reliably. Any updates on this bug?

Manu delgado on Tue, 20 Feb 2018 09:29:48


Are there news in this issue?¿?¿?¿?¿?

Manu delgado on Thu, 22 Feb 2018 16:11:11


We have a very huge issue with this behavior , there are some progress????? 

Connor McMahon on Fri, 23 Feb 2018 01:05:59


It is possible that this is an issue with your triggers not synchronizing. This should happen manually if you are deploying via Visual Studio, but there is a possibility of a bug in that deployment.

See this link about what deployment methods should automatically synchronize your triggers, and how to manually synchronize the triggers yourself.

If your triggers should have been automatically synchronized based on your deployment method, I would recommend manually synchronizing them for now, while adding a GitHub issue here with your site information, deployment method, and when your function stopped working so we can investigate those issues on a case-by-case basis.