Change control — Friday 28 August

The change was a good one: type a part number into search and land on that part, instead of a results page holding a single card. On the test site it does not land on the part. It answers Service Unavailable — the page a customer sees when something is broken.

Before — the live store today/parts?q=CT-NHL-100016 answers 200 with the results grid, and the part is among the cards. q=CT-NHL-100038, q=100412: the same. A word search, q=bearing: the same.

After — the candidate today/parts?q=CT-NHL-100016 answers 503, with no results in the page. So do q=100016, q=CT-NHL-100038 and q=100412. q=bearing still answers 200, so this is not search being down — it is exactly the part-number searches, which are the ones customers actually use.

Re-measured Friday morningUnchanged on the finished candidate — the exact tree the promote carries, not an earlier one. All four part-number searches still answer 503; q=bearing still answers 200; the live store still answers 200 to all five. Thursday's search fix (front#3904) is a different path: it corrected pressing Enter in the search box at the top of the page, which now lands on the part instead of on an unrelated diagram. Opening the results address directly is still broken.

Why it happensThe site works out that one result matches exactly and tries to send the customer straight to that part's page. By the time it decides, it has already begun sending the results page, and a redirect that arrives too late turns into an error instead. The code's own comments describe this exact behaviour having been seen on the test site, so it is a known shape rather than a new mystery.

One thing could not be established, and it decides how serious this is. Every response on both sides came back marked as served from cache, and adding a unique parameter did not force a fresh one, so this cannot tell apart “the test site produces a 503 now” from “the test site produced one earlier and is still repeating it”. The first is a reason to hold the release; the second is a stale page. Someone has to force an uncached render before Friday and say which it is. No number has been invented to cover the gap.

What is not in doubtThere is no issue tracking this. front#3817 is closed as done on the strength of front#3818, which is the change measured above. Whatever the answer, that issue is closed against work whose result has not been confirmed on a real request.

Production todayThe candidate

Production todayThe candidate

The whole index programme is behind four settings, and turning them on is the release, not the merge. Each is one setting, applied without a rebuild, and each is undone the same way. They are listed here in the order they make sense, because two of them are meaningless without the first.

1. A page Google is not invited to read answers “not found” Today a part page carrying nothing still loads as a real page and merely asks Google not to list it. Google reads such a page, decides it is thin, and holds that against the rest of the site. With the switch on, the same address answers not-found outright. Measured: a Ferris part answers 200 with a full page today and 404 on the test site, while a New Holland part that is invited answers 200 on both. So it is selective, which is the thing worth checking.

2. The crawl instructions name each invited partToday the instructions say nothing about parts at all: 13 lines, none about /parts/. With the switch on they hold 4,783 lines — a blanket “do not read parts”, then the 2,384 invited parts named one at a time, in both address forms. This is the one with real consequences if it is wrong, because it tells Google to stop reading everything not on the list.

3. Diagram pages, and what the sitemap offersToday the sitemap offers 2,415 addresses. With the switch on it offers 18,663. That is the number this week's plan quoted as 18,657 — the plan read it from the test site. The live store offers 2,415 today.

4. The master indexing switchNot set on the live store either. Nothing below it can be turned on before it is.

A decision, not a task, and it is John's: turning switch 2 on is the moment we tell Google to stop reading roughly ninety-nine per cent of the part pages it can currently see. That is the intended outcome and it is still the largest single instruction we have ever given it. It should not be turned on in the same hour as the release.

The Kuhn brand page answers today and will answer on Saturday; what it says about itself does not change. Measured: /parts/brand/kuhn answers 200 on both sides and carries “do not list this page” on both. What does change on the test site is that it stops passing Google on to the individual Kuhn parts it lists.

The reason it is withheld is unchanged and is not a decision anybody has taken this week: the Kuhn data takes between twelve and sixty-nine seconds to answer, which slows Google's reading of the whole site. It is written up as front#3666 and it has no owner. This week's plan says the same thing in the same words. Naming an owner is a one-line decision and it has now been outstanding across two releases.

Production todayThe candidate

Before — production today/parts/CT-FER-1650347SM: 200, a 277,856-byte page, carrying noindex, follow and a canonical address pointing at itself, with the full product description Google reads for a real product. In other words: a page that says “do not list me” while looking exactly like a page worth listing.

After — the candidate todayThe same address: 404, 101,782 bytes, noindex, nofollow, no canonical address, and none of the product description. Not a page pretending to be missing — a page that is missing.

The control/parts/CT-NHL-100016 answers 200 with index, follow and its full product description on both sides. That is the row that proves the change is selective rather than blanket, and it is the one to check first if anything looks wrong after a switch is turned on.

Production todayThe candidate

Production todayThe candidate

Production todayThe candidate

If a part page answers not-found, the parts desk still needs to be able to look at it. That is what /lookup/ is: the same part, at an address that exists for people and not for Google.

Before — production today/lookup/CT-FER-1650347SM answers 404. The address does not exist.

After — the candidate todayThe same address answers 200 with the part on it, marked noindex, nofollow and carrying no product description for Google to read.

The part that took four attemptsGetting this right meant walking a narrow line: the twin has to be readable enough for Google to see the “do not list me” instruction, while nothing on the site links to it. Blocking it outright would have been worse than leaving it open — an address Google is forbidden to fetch is an address where it can never read the instruction not to list it. That mistake was made and undone inside this release, which is why several of the lines at the bottom of this page contradict each other in sequence.

Production todayThe candidate

Before — production today13 lines. One Allow: /, ten Disallow lines covering the admin, account, checkout and sign-in areas and the ?highlight= addresses, and one sitemap line. Nothing about parts.

After — the candidate today4,783 lines. The same ten Disallow lines, plus Disallow: /parts/, plus Allow: /parts/brand/ to keep the brand pages readable, plus 4,768 lines naming 2,384 parts one at a time — each in both the full form and the bare part-number form, because the equipment pages link to one and the search results link to the other.

The check that makes this safeThe 2,384 parts named in the crawl instructions are the same 2,384 in the sitemap. Those two lists agreeing is the whole safety property: a part offered to Google in one and forbidden in the other is a page it will fetch and fail on. A new automated check compares them on every change, and it is now part of the gate the release has to pass.

The feed we send to Google Shopping listed products whose page will answer not-found once the switch is on. A product in the feed that leads to a missing page is the shape that gets a feed suspended, so the feed now drops those rows.

The correction inside the releaseThe first version of this dropped rows from the whole feed rather than from the brand section. That was caught and corrected within the same release — both lines are at the bottom of this page, in order, rather than the second one silently replacing the first.

The feed is not being read by anybody today. Merchant Center has never been created, which is on this week's plan as a decision waiting on John, so this change protects something that is not yet switched on. Worth doing, worth knowing.

Part photographs were being loaded from two different addresses depending on where they came from. One was ours; the other was a separate host. The same photograph could therefore have two addresses, be fetched twice, and be counted by Google as two pictures.

Before — production todayOn /parts/CT-NHL-100016: 112 photographs through our own resizing service, 14 from a second host, 2 from our own images folder, and 6 video thumbnails from YouTube.

After — the candidate todayThe same page: 112 through our own resizing service, none from the second host, 5 moved onto our own address, 2 from the images folder, 5 YouTube thumbnails. The second host is gone from the page.

Also in this groupThe site now sends one image format rather than offering two, which halves what has to be produced and stored for each photograph. Brand logos that had been overridden were still being served from the old address; they moved too. And the gallery's picture sizes are now declared up front, which stops the browser guessing.

This is the only group on this page with no switch in front of it, apart from the search problem above. It takes effect on Friday.

The end state everyone is working towards is that crop.clintontractor.net replaces the old www site. Before that can happen, every page on the old site needs a home here, a decided redirect, or a decided drop. Andrew’s site-structure diagram is the list of what those pages are. This is the first block of it, and the only one that needed nothing from anyone outside the team.

Before — production todayService, the service department, booking a service, the credit application, value your trade, schedule a demo, employment, the email club, the showroom, hours and location, staff, testimonials: all twelve answer “not found” on our site and all twelve answer normally on the old one. Anyone following a link, a bookmark or a Google result to one of them landed on nothing.

After — the candidate todayTen of the twelve are real pages answering normally, and Google is invited to read them. The other two go in one hop to the page that already covers them: the service department to service, the showroom to the brands page. The site’s own map for Google grows from 31 addresses to 41 — the ten new pages are the whole of the difference.

Worth saying plainlyThese are our own words on our own pages, not a copy of the old site. Nothing about them depends on a switch, so they are live from Friday morning.

/equipment and /attachments now show the machines and attachments we have for sale, in the same layout as the parts catalogue: the same cards, the same filters, the same search. The listing data behind them is still the snapshot taken from the old site, not a live feed — which is exactly why they carry a “do not index” instruction and are not in the map given to Google.

Before — production today/attachments: not found. /equipment: the parts-diagram hub, a different thing entirely.

After — the candidate todayBoth answer normally and carry noindex. A customer who is sent the address sees the pages; Google is not invited to them until the data behind them is live rather than a snapshot.

What this needs from the callNothing on Friday. The decision to make later is when the listings come from the live source, and only then are they offered to Google.

The showcase above needs the word equipment in the address, and the parts diagrams were occupying it. So the whole diagram tree — the hub, the model index and the model and assembly pages under them, about half a million addresses by the count in the move’s own plan — now lives under /parts-diagrams. Every old address reaches its new one in a single hop, keeping anything that followed the address.

Before — production today /equipment/new-holland-speedrower-130-windrower-tier-3-non-flex-can answers 200 and carries noindex, follow. The live map lists no model addresses at all — only the two index pages above them.

After — the candidate todayThe same address answers 308, one hop, to /parts-diagrams/model/ and the same name, query string intact. The two index pages move with it and keep their place in the map under the new name.

What Google actually sees changeThe addresses, and nothing else. The setting that keeps this tree out of Google is set on the live store today — which is why the live map holds no model addresses and every model page says “do not index”. Friday moves them; it does not open them. On the test site that setting is open, which is why the test map holds 16,250 diagram addresses and the live one holds none. That difference is the switch, not the release.

A part page could be reached at two addresses that differed only in capitals, and the site treated the capitalised one as the real address. From Friday the lower-case form is the real one and the capitalised form goes to it in one hop.

Before — production today/parts/CT-NHL-100016: 200, and it names itself as the real address. /parts/ct-nhl-100016: one hop up to the capitals. The map given to Google holds 2,389 part addresses, 2,389 of them with capitals.

After — the candidate todayExactly the reverse, with the same 2,389 addresses and none of them carrying capitals. One hop, never two.

Why it matters and why it is safeTwo addresses for one page is the oldest way to split a page’s standing with Google between two entries. It is settled here in the only way that keeps every existing link working: the old form still resolves, in one hop, forever.

/catalog, /tires and /favorites redirected into the store. Google’s own record for our site shows no impressions and no clicks on any of them across its full sixteen-month window, against control addresses that returned hundreds in the same measurement. From Friday they answer “not found” instead.

If that turns out to be wrongRestoring one is a three-line change and a deploy. The measurement says nobody arrives that way; if the site’s own error record starts showing one of them after Friday, it comes back.

On a brand's page, the Diagrams tab worked when the first category was selected and did nothing on the others. It now opens on all of them.

Measured/parts/brand/new-holland/diagrams answers 200 on both sides. The page is 5.35 MB on the live store today and 4.17 MB on the candidate — 1.18 MB less to download for the same content.

Worth saying plainlyThis change waited 27 days as a draft. It is on this week's plan under “forgotten requests”, and it is the only item from that list that has actually moved. It is a small fix; the point is the 27 days, not the fix.

Production todayThe candidate

Until now the decision about which brands Google may read lived only in the storefront, and search did not know about it — so search results, the popular-parts lists and the diagram part lists could all offer a part the storefront was withholding. Search now reads the same decision from the same place, and the two agree.

This is the second half of a two-part change, and the halves ship by different routes: the storefront on Friday morning, search by a separate swap. If only one half goes out, the two disagree again — which is exactly the state this change exists to end. Both halves have to go out together, and search goes first.

The old database was switched off on Wednesday morning (v2.3.1). What this release does is remove what was left pointing at it: the driver itself, 31 test settings that no longer set anything, the nightly backup job aimed at a cluster that no longer exists, and a scattering of comments that still described the old arrangement to whoever reads the code next. Nothing here is visible from outside the code.

Why it is on this page at allA comment describing a system that no longer exists is how the next person makes a wrong decision confidently. That is not a hypothetical this week — the plan for switching the old database off was itself checked against a description that had gone stale, and that is the near-miss on the v2.3.1 page.

When a catalogue page cannot be built because something upstream is refusing, the site now answers with an honest “try again” that is explicitly marked as not to be stored, rather than a page that looks fine and is empty. An empty page that looks fine is the worst of the three outcomes, because Google keeps it.

Measured only in part. The marking is confirmed present on the candidate: the failing responses carry no-store on both the browser and the network cache. What could not be confirmed is that it works — the responses came back marked as served from cache anyway. That is the same unresolved observation as the search item at the top of this page, and settling one will probably settle both.

Production todayThe candidate

Ownership lines, test names, code comments and a real work email address used as test data, all still naming someone who left on 20 August. Removed and, where the line stated an owner, replaced with the honest answer — which in two cases is that the lane has no owner.

Worth keepingThis did not come out of the leaving process; it came out of reading the code for something else. The accounts were closed in good order. Thirteen references outlived them, and two of those were the only record of who owned a piece of work.