How the timestamps are fixed

This page explains the preparation of the published data. Start with getting the data if you want to analyse the files, or the audit results for an assessment of timestamp quality. You do not need to repeat these corrections to use a downloaded release.

Why correction is necessary

In an earlier note, I described the SEC's submissions JSON and processed it using R. I switched the bulk processing to Python for efficiency; the resulting Parquet files can still be analysed using R and DuckDB.

Since that note was written, newer submissions data have been found to mix UTC and New York local-clock acceptance times, even when both texts end in Z. Neither that suffix nor the apparent time of day reliably identifies the intended time zone. A UTC clock can fall within ordinary EDGAR hours and still be misinterpreted as Eastern time.

The extractor therefore preserves the original acceptanceDateTime text in filings_raw.parquet. The processor writes interpreted instants to a separate filings.parquet, with columns recording how those instants were established. Keeping these separate allows the interpretation to improve without losing the source text or extracting the JSON archive again.

Use accession-level evidence

The unit of correction is the accession number. The same filing may appear under more than one CIK, but those appearances should represent one acceptance instant. An accepted correction is joined to every occurrence of that accession. A rule for an entire archived CIK file is insufficient because its timestamps can have mixed interpretations.

Previously established instants

Our original analysis had a useful starting point: the May 2024 archive used a consistent New York local-clock convention. The first update compared a Parquet reference derived from that archive with the ZIP downloaded on September 5, 2026. Duplicate-accession comparisons, live JSON, and selected SGML headers supplied further evidence. That ZIP contained 27,209,834 filing rows, with filings dated through September 4.

The second update began with the corrected September 5 Parquet and processed the September 30 ZIP. The released file has 27,272,863 rows; its source ZIP was retrieved on September 30 at 13:39 UTC and had an HTTP modification time of 12:34:49 UTC that day. The release manifest records the source and output checksums.

Readers can use the corrected September 30 reference for their next update without repeating either historical preparation. Thereafter their own prior corrected Parquet can provide the reference; the earlier ZIP need not be reprocessed to carry established instants forward.

Only supported prior instants can act as anchors. Usable prior rows for an accession must agree on one instant, and interpreting the new raw clock as UTC or America/New_York must reproduce it. An unresolved fallback is not an anchor, although its reconstructed source clock may supply one member of a new exact clock pair. Non-timestamp fields come from the new archive.

New candidates retain the original text in acceptanceDateTime_raw as well as in the separate raw table. A source clock remains eligible for pair matching even if its previous interpretation was unresolved or marked historical_midnight_unverified. Earlier raw snapshots can also be supplied with --historical-raw. This prevents an unresolved label, or an intermediate release that lost the earlier clock, from suppressing an otherwise exact pair.

Differing clocks for the same accession

Duplicate rows, prior snapshots, and live JSON may supply two distinct clocks for one accession. The useful case has exactly two valid clocks that resolve to one instant when the earlier is interpreted as America/New_York and the later as UTC. For example, 08:01:07 in New York on a December date and 13:01:07 UTC on that date describe the same instant.

The package tests equality using named time zones, accounting for daylight saving and changes of date. A difference of four or five hours alone is insufficient. Neither the newer snapshot nor live JSON is assumed to supply the UTC member. Equal clocks provide no new timezone evidence, and unexplained differences remain in diagnostics. Outside-hours status is not required.

An exact, uncontradicted Eastern/UTC pair is sufficient to regard the accession as resolved; an SGML lookup is not required. SGML remains useful for independent audits and investigating conflicts. Any observed SGML contradiction must be reported rather than hidden by the pair-based classification.

Direct SGML observations

A filing header's <ACCEPTANCE-DATETIME> provides accession-level evidence, interpreted here as a New York local clock and converted using America/New_York. Exact SGML observations take priority over weaker file-level inference; disagreements remain visible in the diagnostics.

Missing timestamp tags and failed requests are retained separately. Transient failures can be retried. When the tag is absent, the audit uses the filing index's Accepted timestamp as ground truth, interpreting it as Eastern. The source and its missing SGML tag remain visible. Literal midnight alone is not a sufficient rule for declaring a timestamp missing.

Historical date-only values

Older filings established to have midnight values and no SGML acceptance tag are resolved as date-only. Midnight in America/New_York can represent their date in a timestamp column, but is a storage convention, not an observed acceptance time. Date-only resolution must remain distinguishable from a resolved acceptance instant and must not supply intraday evidence.

In the September 30 audit, 763 of the 770 missing-tag rows had raw clocks of 00:00:00 (612 rows), 04:00:00 (59), or 05:00:00 (92). All 763 matching records in the original May 2024 ZIP had 00:00:00 clocks. The later 04:00:00 and 05:00:00 values each correspond to midnight Eastern on the relevant date. These cases support the date-only classification; midnight alone, without the historical and missing-tag evidence, does not establish it.

This interpretation refines the status of those historical values. The frozen September 30 release retains its original provenance labels. The supplemented audit also identifies 52 midnight-storage discrepancies in that file; date-only agreement must not be presented as exact timestamp agreement.

Provisional live-file inference

One live JSON response can supply clocks for many accessions. Supported prior instants, exact observed clock pairs, and SGML observations can suggest the convention used by that response. If the available anchors agree, the processor can use that convention to propose timestamps for its other observed clocks. These proposals are provisional, not confirmations.

The October 2 audit found two counterexamples to a uniform response convention: a response can contain both Eastern and UTC clocks. An inferred timestamp must therefore never become a confirming anchor, even when inherited from a previous Parquet file. Newly processed rows retain such guesses with provisional_live_file_inferred provenance and are counted separately from supported rows. The older live_file_inferred label is also treated as provisional when reading a predecessor.

File conventions remain useful for identifying cases to investigate. The live collector includes provisional rows, and the processor compares actual source clocks across snapshots and live responses. If two observed clocks satisfy the Eastern/UTC pair rule, the pair supplies the resolution; no SGML lookup is needed for that rule. A converted provisional guess is not a second observed clock. Identical source clocks do not establish a pair.

Diagnostics preserve disagreements between provisional predictions and resolved instants in provisional_pair_discrepancies.parquet, with the resolving provenance. SGML disagreements are included as well. This makes the hypothesis useful for finding errors without allowing it to confirm itself.

The processor also exports mixed_live_reference_queue.parquet. It contains unconfirmed accessions from response versions already shown by direct evidence to mix interpretations. Iterating this selection to closure required three rounds containing 10,738, 343, and 17 accessions. All 11,098 live clocks matched their ZIP clocks, while SGML classified 11,044 as UTC and 54 as Eastern. The Eastern observations were confined to late August and September 2026 and clustered by source file. Thus matching clocks are useful targets in a known-mixed response, but remain non-evidence until a direct reference is collected.

Selecting additional checks

The initial candidate identifies gaps in the available evidence. Screening unresolved and provisional recent filings is preferable to looking only outside EDGAR hours, because an incorrectly interpreted UTC clock need not fall outside that window. Live JSON comparisons should precede a large SGML queue: one response can supply actual clock pairs for many accessions in one request.

Remaining questions include missing live matches, contradictory evidence, and live response versions without anchors. For the September 30 update, the follow-up checked one accession in single-accession response versions and the first and last accessions in multi-accession versions. Of 23,906 queued accessions, 23,905 supplied a timestamp tag and one returned 404. These selected checks did not establish a uniform convention for every filing. The subsequent probability-sample audit found two exceptions.

EFFECT was excluded from the outside-hours screen. The targeted anchor collection also excluded CORRESP, UPLOAD, and DRSLTR. These forms can still participate in exact clock-pair comparisons and the random audit; their names or a global date cutoff should not determine their time zone.

Preserve uncertainty and provenance

The released file records timestamp_provenance, timestamp_reference_id, and timestamp_interpretation. For example, sgml denotes direct header evidence and provisional_live_file_inferred an unconfirmed response convention. Rows without an accepted instant retain an Eastern interpretation labelled unresolved_raw_as_eastern: a compatibility fallback, not a verified timestamp.

The main Parquet carries established instants and their status into a later update. Separate observations retain source clocks, retrieval versions, failed attempts, and evidence for accessions absent from a later snapshot. Diagnostics record inputs, support counts, rejected evidence, and disagreements. A temporary DuckDB database performs the computation; it is removed after the candidate and Parquet diagnostics are exported.

Request completion, evidence-supported coverage, and an accuracy estimate are different things. The audit results assess a frozen candidate with a fresh probability sample rather than treating successful collection as proof of correctness. The advanced workflow gives the commands for readers who want to prepare their own update.