Check that a daily table or file is fresh
The daily load ran on time and loaded the usual number of rows, but the newest date in the table is from last Tuesday: the source sent the same extract again. Row counts and exit codes don't notice this. Checking the newest date on every run turns that into an alert the same morning.
Why stale data passes every other check
A stale load looks healthy from the outside. The job started and ended on time, it exited with 0, and because yesterday's file had 48,000 rows, so does today's copy of it. Typical causes:
- an upstream export failed and left the previous file in place;
- an incremental load filters on "since last run" and the watermark didn't move;
- a view or extract is cached or reads a snapshot that stopped refreshing;
- a partner sends the file late, and your job picked up the old one first.
What tells you the truth is the data itself: the newest date in it. A freshness check compares that date with today.
Send the newest date with each run
Add last_date to the finishing ping, as YYYY-MM-DD. When you tick Latest date on a job's Code tab, the code sends the maximum of your date column and TaskBeacon adds the rule last_date is a date older than 1 day. In Python:
# --- TaskBeacon: top of the script (standard library only) ---
import json, traceback, urllib.request
TB = "https://taskbeacon.io/ping/your-job-id"
def tb_ping(path="", data=None):
body = json.dumps(data).encode() if data else None
req = urllib.request.Request(TB + path, data=body, headers={"Content-Type": "application/json"})
try:
urllib.request.urlopen(req, timeout=10)
except Exception as e:
print("TaskBeacon ping failed:", e)
tb_ping("/start")
try:
# --- your code here ---
from datetime import date
df = [{"date": date.today().isoformat()}] # example data: replace with your own (e.g. pandas.read_csv)
except Exception as e:
# the error and the last 20 lines of the traceback go with the failure
tb_ping("/fail", {"error": f"{type(e).__name__}: {e}"[:200],
"log": "\n".join(traceback.format_exc().splitlines()[-20:])[-4000:]})
raise
# --- TaskBeacon: end of the script ---
tb_ping("", {"rows": len(df), "last_date": str(max(df["date"] if hasattr(df, "columns") else (r["date"] for r in df)))[:10]})In the app, the Code tab of your job gives you this code with the job's own address instead of your-job-id, and you can untick the parts you don't need.
Replace the example with your data and "date" with your column (for a pandas DataFrame, df["date"].max() is what gets sent). The age is counted in your time zone from Settings, so a table that is filled with yesterday's data every morning should get the limit 1; one that is filled on weekdays only, a limit of 3 to cover Mondays. Change it on the job page under Alert me when.
Freshness rules are part of Pro, open to everyone for now; see Pricing.
What you get
- Odd data when the newest date is too old, with the reason in one line, for example last_date is 2026-11-03, 6 days old (limit 1).
- The row count in the same ping keeps its own rules, so an empty load and a stale load are both caught; see 0 rows or far fewer than usual.
- Missed when the load didn't run at all, which no check inside the database can tell you.
For files that a partner or another team drops into SharePoint or OneDrive, TaskBeacon can also watch the file's modification time directly, without any code: see a file that didn't arrive by 8:00.
Without TaskBeacon
In a dbt project, declare freshness on the source with loaded_at_field, warn_after and error_after, and run dbt freshness (the older dbt source freshness is now called legacy) as part of a scheduled job. That works well, with one gap: the check itself has to run. If the job that runs it doesn't start, nothing warns you; see dbt Cloud job alerts. Outside dbt, a SQL query such as SELECT MAX(load_date) FROM sales in a scheduled script that raises an error when the date is too old does the same.
Connected Claude or ChatGPT to TaskBeacon? Ask it "what failed last night, and why?" and it answers from your jobs. How that works.
Related
In the Docs: Latest date · Statuses · Python.
- Get an alert when a job loads 0 rows, or far fewer than usualExit code 0 says nothing about how many rows arrived.
- Get an alert when a file doesn't arrive in SharePoint by 8:00Triggers fire when a file arrives; nothing fires when it doesn't.
- dbt Cloud job failure alerts, and runs that never starteddbt notifies about runs; a run that never started sends nothing.
TaskBeacon is free to start, with every feature open for now and no credit card. Create a job, paste the code, and the first ping shows up on the job page.
Start free View live demoSources
- dbt docs: freshness (loaded_at_field, warn_after, error_after; dbt freshness replaces dbt source freshness) (updated 2026-10-12)
- Reddit r/dataengineering: What breaks first in small data pipelines as they grow? (about 2025 (estimated))
Page checked against its sources on 2026-11-09. Menus and limits of other products change; if something here no longer matches, write to support@taskbeacon.io.