The Deploy That Kept Failing
Five production deploys. Five different failures. One working system at the end.
This is the story of a cutover day where everything that could break did break — and each failure taught us something about how CI/CD pipelines lie to you.
The Setup
We had a SaaS marketplace application ready for production cutover. Infrastructure was Terraform'd. Code was deployed. The pipeline was "working." All that remained was pressing a button in a partner portal to redirect real customer traffic to our new stack.
Simple, right?
Failure #1: The Silent Deploy
The first deploy appeared to succeed. The CLI tool uploaded a 31 KB source archive, printed "remote build started," then exited 0. Success.
Except the remote build never actually completed. The function was still running last week's code. The CLI had reported success on what was essentially a no-op.
What I learned: Exit code 0 means nothing if you don't verify what's actually running. We added post-deploy verification — hit the deployed function with a known-bad input and check that you get the specific error your code returns, not a generic platform error.
# Our handler returns 400 with "invalid_contract" for empty input.
# A broken deploy returns 500 with a generic Azure error.
if [[ "$HTTP_CODE" == "400" ]] && echo "$BODY" | grep -q "invalid_contract"; then
VERIFY_OK=true
fi
Dumb? Maybe. But it catches the failure mode that the official CLI doesn't.
Failure #2: The Binary Mismatch
The remote build was flaky, so we switched to local build. Compiled everything locally, pushed the zip. Deploy succeeded (for real this time). Function started. Then immediately crashed.
ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found
The local build compiled a cryptography package against a newer glibc than the cloud runtime supports. The binary was platform-incompatible. We'd traded a flaky deploy for a broken one.
What I learned: "Build locally, deploy remotely" only works if your local environment matches the target. For interpreted languages with native extensions, it often doesn't. We reverted to remote build and added the verification step to catch silent failures.
Failure #3: The Naming Problem
With the deploy fixed, the test suite ran. Smoke tests failed immediately:
Plan 'hosted-data-services' not found on portal
The test was checking plan names against a staging environment, but the production and staging environments use different plan name conventions. When we redirected tests to check staging plans during production deploys, we forgot to also swap the plan names.
What I learned: Environment-specific values are a dependency graph, not a flat list. Swapping credentials without swapping the data those credentials give you access to is half a migration.
Failure #4: The Guard That Didn't Know About "Skip"
We had three modes for portal write tests: staging, production, and skip. The integration test runner had a safety guard — a hard check that blocks tests from accidentally writing to the wrong portal.
The guard knew about staging (allow) and production (allow if explicit). It did not know about skip. When we set the mode to skip for production deploys, the guard saw a non-staging portal address, panicked, and killed the pipeline.
# The guard checked two conditions but missed the third
if [[ "$WRITE_TESTS" != "production" && "$PORTAL_ADDRESS" != "staging..." ]]; then
echo "ERROR: Integration tests only run against staging portal"
exit 1
fi
What I learned: Every valid input must have a handler. The variable had three values, but one code path only handled two. We renamed the variable from the confusing WRITE_TESTS to PORTAL_WRITE_TARGET with values none, staging, production — and added the missing case.
Also: this failure should have been catchable locally. The exact same script runs in CI and on a developer machine. We could have run PORTAL_WRITE_TARGET=none ./scripts/test/run.sh production integration and seen the error in two seconds. We didn't, because we only thought about what changed, not what the change affected.
Failure #5: The Race Condition
Deploy #5. Everything passes — smoke, integration, unit tests. Deployment starts. Gateway deploys. Landing page deploys. Handler... fails.
Error Uploading archive... (Conflict).
Server Response: Run-From-Zip is set to a remote URL.
Deployment is not supported in this configuration.
A previous broken deploy (Failure #2) had set an app setting that conflicts with the normal deploy method. The CLI tried to remove it, then immediately tried to upload — but the removal hadn't propagated through the platform's API yet. A race condition between "delete this setting" and "now deploy."
We checked: the setting was already gone by the time we investigated. The CLI's own cleanup had worked, just not fast enough.
What I learned: Distributed systems have propagation delays. "I deleted it" and "it's deleted" are not the same statement. The sixth deploy succeeded with zero changes — just timing.
The Sixth Deploy
Deploy #6. All green. All twelve pipeline jobs passed. Handler deployed, verified, responding correctly. We ran a test purchase through the actual marketplace. Subscription created. Tenant provisioned. Email with credentials delivered.
We pressed Go Live.
The Meta Lesson
Each failure was individually simple. None required more than a few lines to fix. But the sequence matters — each fix revealed the next failure, which was invisible until the previous layer worked.
This is why scripted pipelines matter. Not because they prevent failures, but because they make failures reproducible. Every command that ran in CI could have run on a laptop. The pipeline is just a wrapper that calls scripts with parameters. When it breaks, you reproduce locally, see the error, and fix it.
The moment your CI does something you can't reproduce locally, you've lost the ability to debug it. That's the real lesson from a day of five failures: keep the pipeline dumb, keep the scripts smart, and verify everything.
Total time from first failure to Go Live: about 6 hours and 19 PRs. The actual code changes across all fixes? Probably under 50 lines.