Stop 5 of 10 · Weekly
The monitoring agent
Forty sites checked every night as an anonymous visitor, a diff sent to Claude for diagnosis, findings written to the studio database, and the list of things the agent may never do.
Forty sites, every night, as a stranger
Dany's agent is a Node script on the small host. It knows nothing about any client system. It fetches pages the way anyone with a browser could, records what it found, asks Claude what changed, and writes findings to the studio database. That is the whole job.
What it checks per site
Four pages each: home, the main service page, the contact page and the sitemap. Per page it records status code, time to first byte, final url after redirects, page bytes, title, canonical tag and the count of internal links. Per site it records TLS days remaining and whether robots.txt is reachable.
Nothing here needs a login, so the agent has none.
The must-nevers
| Never | Because |
|---|---|
| Any method except GET and HEAD | A POST is a form submission, and a form submission is a real enquiry in somebody's inbox |
| Log in, or hold any client credential | The agent has no account anywhere, so no account can be misused through it |
| Fetch a cart, checkout, logout or admin path | Those paths change state on a live site even under GET |
Ignore robots.txt or run parallel requests at one host |
You are a guest on a host you did not pay for |
| Send page HTML to Claude | Customer data hides in confirmation pages and review widgets |
Its user agent names Dany's studio and carries a contact address, so a host administrator who sees the traffic can ask rather than block.
Rate limits and retries
One request at a time per host. Two requests a second across the whole run. Retry only on 429, 502, 503 and a network error, three attempts, waits of roughly one, four and sixteen seconds with jitter added, and Retry-After obeyed when the header is there. Five consecutive failures at one host opens a circuit breaker and the rest of that host's sites are skipped with a note. The run has a hard budget: no more than four hundred requests and no more than twenty minutes.
Write me a Node module, agents/fetch-page.mjs, that fetches one url and returns status, timing, final url, byte count and the response headers. It must use GET or HEAD only, follow at most three redirects, time out at ten seconds, and retry only on 429, 502, 503 and network errors with three attempts and exponential backoff with jitter. Honour Retry-After when present. Take a per-host queue so two requests never run at one host at the same time, and a global limiter of two requests a second. Set a user agent naming my studio with a contact address. No cookies, no credentials, no request body ever. Explain each limit in one comment line, and tell me which of them you would set differently for a shared host.
Here is one site's check rows for the last four nights and the response headers from tonight: [paste the four rows and the headers]. Tell me what changed and what the most likely cause is, ranked, with the evidence in the rows for each one. Say clearly when the rows cannot distinguish between two causes and name the one extra reading that would. Do not suggest anything that requires logging in to the site. Finish with a one-line finding, a severity from 1 to 4, and the owner: the studio, the host, or the client.
Write a node --test file that starts a local HTTP server on 127.0.0.1 with four routes: one healthy, one returning 500, one that sleeps four seconds, and one redirect chain four deep. Point my checker at it and assert the findings it produces for each. Then assert that across the whole test run the fixture server received no request with a method other than GET or HEAD, and no request carrying a cookie or authorization header.