The Michigan Lakes Case Study

Lake Finder answers one question: which lakes near me are worth fishing, and what does the state know about them? Michigan publishes almost every fact that question needs. It publishes none of them together.

OpenStreetMap knows where the water is and what people call it. The DNR knows depth, boat access, which species have been documented, what has been stocked and which waters hold state records, spread across half a dozen ArcGIS services. Depth maps are not a service at all: they are a directory of scanned PDFs, filed by county, with no index and no API in front of them. Nothing joins any of it, and none of the sources share a key.

So the app is the join, and the join is computed on every request. One Cloudflare Worker invocation, a hard deadline, a capped number of outbound calls, and a dozen ways for a public service to be slow rather than down. This is what the interesting half of that turned out to be: not the fetching, which is easy, but deciding what the app is allowed to claim when a source does not answer.

Every search fans out to these. The last column is the part worth designing for: what the reader should see when one of them is having a bad afternoon.

The third row is the one that shapes the rest of this page. You cannot ask the archive what it has. You can only ask whether one particular file is there, one lake at a time, by building a path out of a county name and a lake name and reading the status code that comes back. Everything downstream is a consequence of that.

An existence check per lake is cheap on its own and ruinous in bulk, so the budget is declared up front and everything else is sized against it.

const PER_CALL_TIMEOUT_MS = 10000;
// A large radius covers a lake-dense area, so Overpass needs longer to answer.
const OVERPASS_TIMEOUT_MS = 14000;
const TOTAL_BUDGET_MS = 26000;

// Selectable search radius (miles) -> the most mapped lakes we show.
const MILES_LIMITS = { 15: 12, 25: 16, 40: 20, 100: 24 };

// How many nearby candidates we CHECK before filtering to the ones we show.
// Larger than the display cap so a cluster of nearby mapless ponds can't
// crowd the real lakes out of the results.
const CANDIDATE_LIMITS = { 15: 18, 25: 22, 40: 24, 100: 28 };

// Only this many still-unmapped lakes get their own county geocoded.
const DNR_COUNTY_FALLBACKS = 6;

Two of those pairs are worth pulling apart. The candidate cap sits above the display cap deliberately: filtering happens after the archive is consulted, so checking exactly as many lakes as you intend to show means one cluster of unmapped ponds can push every real lake off the list.

The county lookup is two-phase for the same reason a database query gets an index. Nearly every lake in a search shares the origin's county, so that county is geocoded once and tried against all of them with nothing but a cheap existence check each. Only the nearest lakes still without a map get their own county resolved, they may sit across a line, and only six of them, and not even those once the display cap is already full. The version without that bound geocoded per lake and timed out into an empty list on exactly the dense, lake-rich searches the app exists for.

The existence check wants one thing from the archive: a status code. It asks for a single byte to avoid pulling down a scanned PDF it has no use for, reads the status, and moves on.

In production that jammed the whole request. A Response left unread holds its connection open, and the runtime caps how many can be open at once. Every candidate lake was probed concurrently, every probe kept its connection, the cap filled, and the runtime started killing the oldest to get out of it. The log line, nine times in a single request: a stalled HTTP response was canceled to prevent deadlock. Every other upstream call in that request queued behind the pile.

/* Let go of a Response whose status was the whole answer.
 * Any code path that decides against a Response without reading
 * it owes this call. Never throws: by the time it runs, the
 * status is already in hand. */
export async function dropBody(res) {
  try { if (res && res.body) await res.body.cancel(); }
  catch (e) { /* nothing left to protect */ }
}

With that, the probe reads as it should: ask for a byte, keep the status, hand the connection back. Note what the status codes are allowed to mean. A 404 is the archive telling you there is no map for this lake, which is a fact. Anything else is the host having a problem, which says nothing about the lake at all, and the difference between those two is section 5.

async function urlExists(url, deadline) {
  if (timeLeft(deadline) < 1500) return null;
  try {
    const res = await fetchWithTimeout(url,
      { headers: { "User-Agent": UA, Range: "bytes=0-0" } },
      Math.min(5000, timeLeft(deadline)));
    await dropBody(res);
    if (res.status === 200 || res.status === 206) return true;
    // A 404 is the archive telling us there is no map. Anything else
    // is the host having a problem, which says nothing about this lake.
    return res.status === 404 ? false : null;
  } catch (e) {
    return null;
  }
}

Three return values, not two. true and false are answers; null is the absence of one, and it is a different thing from false in every way that matters to the reader. Most of the bugs in this route have been some version of collapsing those two.

"No lakes near Grayling" is a claim, and the search has to earn it. The same empty array is produced by a genuinely quiet stretch of map and by four services declining to answer, and those two owe the reader completely different sentences.

I have written this guard three times. The first two were wrong in the same way, and the way is worth more than the fix:

The gap that actually shipped is a good illustration of why enumerating failures does not work. The enrichment step is wrapped so a thrown error becomes a zeroed tally, which is sound, except that a wholesale failure then reports checked: 0, found: 0, unknown: 0 and a search that genuinely found nothing reports exactly the same three zeros. unknown > 0 was false. Eighteen candidates went out as an empty lake district, and a debug run minutes later showed eleven of them had depth maps.

So the third version asks whether the search is entitled to its answer, and treats anything less as an outage:

if (!lakes.length) {
  // Could the water sources be read at all? One reachable and
  // genuinely quiet is a real answer; neither reachable is the
  // double outage.
  if (!lakesResult.read && hydroDiag.queryOk !== true) {
    return { success: false, error: "Couldn't reach OpenStreetMap or the "
      + "Michigan DNR's water data, so there's nothing to search yet." };
  }

  // There were candidates, so the depth-map gate is what emptied the
  // list, and it only gets to do that on candidates it RESOLVED.
  if (candidates.length &&
      (mapTally.unknown > 0 || mapTally.checked < candidates.length)) {
    return { success: false, error: "The Michigan DNR's lake-map archive "
      + "is not answering, so depth maps could not be confirmed." };
  }
}

mapTally.checked < candidates.length is the load-bearing clause, and it is the one no counter can report on its own. It does not ask what went wrong. It compares what the stage claims it did against how much work there was, so a stage that never ran is caught by arithmetic rather than by having been anticipated. That generalises: when a failure and a legitimate zero produce identical counters, stop reading the counters and find something outside them to check against.

Guarding the empty case still leaves the quieter problem: a short answer that is not empty. Ten lakes when there should be twenty is invisible from outside, and it is invisible from the inside too unless the response says what it started with.

So every answer carries a small record of what was considered and where it went. Deliberately not a debug block. Debug bodies are opt-in, skip the cache, and get read by somebody who already suspects something; the entire problem here is that nobody suspects anything.

emptiness: emptiness(candidates.length, "OpenStreetMap water bodies", {
  noDnrMap:     /* checked, and the archive has no map */,
  mapUnchecked: mapTally.unknown,
  overCap:      /* mapped, but past the display cap */,
})

The first two are kept apart on purpose, and merging them is the mistake this record exists to prevent. "Eighteen candidates, eighteen without a map" reads as a lake district nobody has ever surveyed. It reads identically when the archive refused eighteen times. Two facts, two different fixes, two different sentences owed to the reader, and one number if you let them merge.

That is the real route, called live, drawn from the record on its ordinary cached response. Whatever it says today is what you are looking at: a full list draws the funnel that produced it, an empty one draws the funnel that emptied it, and an outage prints the guard's own sentence instead, which is section 5 working rather than section 5 failing.

A route this conditional decays quietly, so most of the maintenance cost is paid in machinery that fails loudly instead.

The thread through all of it is the same one sentence, which turned out to be the actual lesson of building on public data: an app is allowed to say what it knows, and it is allowed to say that it does not know, and the expensive bugs are all the third thing, where not knowing gets rendered as knowing.

The app is at Lake Finder, the upstreams it depends on are at Status, and the shape of the site around both is at How This Site Works.