# Data Lakes now show which files were not ingested, and why

Drive and GitHub lakes now list the files a sync did not ingest, with names and reasons. This post also covers GitHub lakes that follow the repository, the access-lost state, Drive sync status, a source-first create flow, research presets, and how lakes behave in chat.

Data Lakes now show which files were not ingested, and why Data Lakes now show which files were not ingested, and why Bike4Mind Engineering · October 14, 2026 A data lake in Bike4Mind is a connected source, such as a Google Drive folder or a GitHub repository, that is synced and ingested so chats and research can draw on its contents. Most of the time that works quietly, which is exactly the problem. If a file never made it into the lake, nothing looks wrong until you ask about it and the answer comes back confidently incomplete. This post covers the recent Data Lake work in the open-source core, from connectors to research to chat. Most of it serves one idea: a lake should be able to tell you what it actually contains, and what it left out. Files that were not ingested Sync has always skipped files it could not ingest. What changed is that the skips are now recorded and shown. When a Drive or GitHub lake has any, the lake manager shows a chip such as 3 files not ingested . Click it and you get a dialog listing each file by name, its path when that differs from the name, the reason, and its size when the source reported one. The chip and the list are visible to people who can manage the lake, since the rows name files in the connector owner's source that never became lake members. The list is meant to read as "what is missing right now". A file leaves it when it later ingests or disappears from the source. An edited file that fails a check gets an entry even though its previous version is still in the lake, because the current version is what is missing. Disconnecting the source clears its entries. The reasons, as the code defines them There is one shared list of reasons, and the UI copy for each is typed against it, so adding a reason without wording for it fails typecheck. Today there are nine: oversized : larger than the per-file size limit. The size is known from the listing, so the file is skipped before any bytes are downloaded. exceeds_max_file_size : larger than the maximum file size an admin set for the workspace. oversized_after_fetch : the listing understated the size, and the downloaded file turned out to be over the limit. unsupported : a Drive item with no download or export path, such as a Google Form or a shortcut. export_too_large : a Google-native document whose export exceeds what Drive allows. error : the download failed for a reason other than rate limiting. The dialog says it will be retried on a later sync. storage_limit : the lake owner's storage is full. It is retried once space is freed. binary : a GitHub file that is not text. invalid_utf8 : a GitHub file whose bytes are not valid UTF-8. Two of the numbers are fixed. Drive files over 50 MB are skipped, and GitHub files over 1 MB are skipped, unless the workspace's admin file-size setting is lower, in which case that setting is what you hit and the reason says so. A GitHub file counts as binary if there is a null byte in its first 8 KB. One limit of the list deserves saying plainly. For GitHub, the sync also filters by extension and ignores whole categories on purpose: node_modules , dist , build and similar directories, lockfiles, symlinks, and extensions outside an allowlist. Those are rules about what a lake is for, not failures, and they do not appear in the list. Only oversized files are recorded at that stage. If you expected a particular .dat file to be searchable and it is not there, it will not be on the not-ingested list either; the extension allowlist is the thing to check. GitHub lakes that keep up with the repository A GitHub lake is connected through a GitHub App, read-only, from the lake manager or from the create wizard. The connect flow is authorize-first: you authorize, then pick the repository from a picker, and you land back where you started. A lake can wait on one connector at a time; the server enforces one connector per lake across Drive and GitHub, claiming the lake atomically before the connection is created. Staying current used to depend on pressing Re-sync. Now a signed push to the repository's default branch queues a re-sync through a webhook. Unsigned deliveries, other branches, tags, branch deletes and non-push events do nothing, and every ignored delivery is logged with a reason code. Because webhooks get missed, a scheduled reconcile compares each repository's default-branch HEAD with the last synced commit and re-enqueues when they differ. It will not re-enqueue the same target commit again for six hours, so a sync that keeps failing does not loop. The reconcile is off by default: it runs only on stages where it is scheduled and while an admin flag is on, and elsewhere the next push or a manual Re-sync catches up. While a lake syncs, the panel shows the repository and branch with a Syncing chip, then "N of M files indexed" and a progress bar (just a running count until the tree has been read), plus the last synced time and commit. Beside it, "N files synced, M skipped" appears, and the skipped count opens the sync rules. The sync rules live in the common package, so the client shows the same rules the ingest enforces. If the deployment has no GitHub App configured, the connect route answers 503 with a plain message (GitHub App not configured) instead of failing opaquely. Access lost GitHub can take access away in two ways. If it tells us, the App's webhook handles an uninstall or a repository being removed from the selected set, and queues a revoke that tears the connection down and purges the files it ingested. If we only find out because a sync fails, the connection gets an access_lost status and the lake is left as it is, for the case a retry cannot fix: GitHub answered that the App can no longer read the repository because it was uninstalled, removed from the selected set, or deleted or moved. The lake shows "Access lost to owner/repo" instead of a generic error, states that files already in the lake are untouched (until someone chooses Disconnect, which deletes them), and offers Fix on GitHub (linked to the installation, not an owner-only settings page) and, for managers, Disconnect. Once access is restored, Re-sync picks up where the lake left off. Drive lakes The Drive lake panel shows when the lake last synced and how it stays current. The first line reads "Last synced" with a relative time, or "Last sync attempt" if the last run recorded an error, since a failed run also stamps the time. Next to it is one of three schedule messages: it syncs automatically about every N hours, automatic sync is paused, or it syncs only on a manual Re-sync. A connection is re-polled at most every six hours from its last actual sync, and the text comes from the same code that runs the schedule. Several fixes make Drive failures visible instead of silent. A sync whose claim has gone stale is reported as stalled rather than showing "Syncing" forever. If the connected folder is unshared or trashed, ingest now probes it first and parks the lake in needs_reconnect with an error naming the cause, because Drive's changes feed would otherwise report a healthy empty delta. Re-sync reuses the stored folder without reopening the picker, and a dead grant offers Reconnect Google Drive. A personal lake, not only an organization lake, can now connect a Drive folder; on a personal lake only its creator can. And a file that is trashed or moved out of the folder is deleted along with its chunks and stored object, so it is no longer searchable or counted against storage, where before it was only unpicked from the lake. That delete only runs when Drive positively confirms the removal. A bare "removed" signal, which Drive also sends when you merely lose access to a file, only unpicks it. A copy that is shared, belongs to another lake, or has no owner left is unpicked but kept. Source first when creating a lake Creating a lake now starts with the question that decides everything else: where does the content come from? The first wizard step offers three cards, Upload files, Google Drive and GitHub repository. Picking GitHub shows a panel that creates the lake and sends you to GitHub; the wizard also resumes correctly after the Google Drive OAuth redirect. A lake born from a connector is created connector-fed, so it will not accept the wrong kind of content later. GitHub is organization-only and limited to org owners and managers, and the card says why when it is unavailable. Research A lake's research configuration decides how hard a run looks for new content to propose, and how much it may spend. The form now has three presets that set the judge model, depth and cost ceiling together: Quick : up to 10 search results, 5 proposals, a cost ceiling of $0.05, and the default relevance judge. These equal the old form defaults, so someone who never opens the advanced section saves what they saved before. Standard : 25 results, 10 proposals, $0.25 ceiling, default judge. Deep : 50 results, 25 proposals, $1.00 ceiling, and a different judge model, GPT-4.1 instead of the default mini variant. Research runs also fail more honestly. If the relevance judge fails three times before scoring anything, the run stops with a judge_unavailable reason instead of paying for a judgment on every remaining candidate, and the unreached candidates are counted as not judged so the run card still adds up. A failure after the judge has worked is treated as a blip. Web search for research now uses the same lead-and-backup hedge as chat, so one slow provider no longer fails the search when another is configured. Chat, drafts and tools The lake picker and the scope strip now flag lakes that chat retrieval cannot search, and the scope strip lists them under a "Not searched" heading. Starting a chat on a draft lake asks first ("This lake is a draft"), with Publish and start chat for managers and Start anyway for everyone, because a draft does not ground answers and users were assuming the upload had failed. Chats started on a lake leave your own and shared files out by default, and a My files chip in the scope strip turns them back on. The assistant can list your writable lakes, create a lake and save generated content into one; new lakes it creates are always drafts. The data lake API is part of the public API contract: list lakes, get a lake, search a lake, and get, add or remove a member file. Under the hood Skips are stored in their own collection, keyed per lake and source rather than per sync batch. That choice matters: every sync creates a new batch and finished batches drop out of reads, so a batch-scoped list would vanish at the next sync. Each row is unique on lake, source kind and source id, which is the Drive file id or the repo-relative path for GitHub. The repository's recordSkip upserts, so a file skipped on every full walk stays one row and keeps its first-seen time. The two ingest workers write the rows. The Drive ingest handler records a skip at each gate (size, admin limit, unsupported or failed fetch) and clears the row when the file ingests. The GitHub slice handler records oversized files from the tree listing and later skips such as storage-limit and oversized-after-fetch, again clearing rows on success. A sync that saw the whole source prunes rows for files no longer present, and the contract says not to do that from an empty listing, so a transient empty response cannot wipe the list. The UI reads rows through GET /api/data-lakes/:id/skipped-files , which requires write access to the lake, takes limit (default 50, max 200) and offset , and returns the page with an exact total and a hasMore flag. Rows are ordered by first-skipped time, which a re-skip never changes, so a sync that refreshes every row between two page loads does not reshuffle paging. If a file should have ingested and is not on the list, open an issue on the repository. All of this is in the open-source Bike4Mind repo: github.com/Bike4Mind/bike4mind .