globusfs: an fsspec Filesystem for Globus Collections

Building an fsspec backend for Globus so pyarrow, pandas, and dask can read a collection the way they read s3:// — and the HTTPS-server quirks that made it harder than it should have been, including a 404 that means two different things.

Globus is how large scientific datasets actually move between facilities. It is not, however, something your data stack can read. There is no fsspec backend for it, so pyarrow, pandas, dask, and grain cannot open a Globus collection the way they open s3:// or gs://. You stage the whole file first, then work with it.

globusfs closes that gap: one backend, all of those clients.

import globusfs

# Browser login once; tokens persist to ~/.globusfs/tokens.json
fs = globusfs.filesystem("<collection-uuid>")

fs.ls("/")
with fs.open("data/file.parquet", "rb") as f:
    ...

The payoff is column projection over the wire. Reading one column of sixty from a remote parquet file transfers a few KB instead of 360 KB, because the reader issues range requests for exactly the bytes that column needs:

import fsspec, pyarrow.parquet as pq

fs = fsspec.filesystem(
    "globus",
    collection_id="isaac",
    https_url="https://g-05a4b6.2d513.8443.data.globus.org",
)

with fs.open("isaac/ability/ALL_2007-01.parquet", "rb") as f:
    table = pq.ParquetFile(f).read(columns=["author"])

Two services, because neither is enough

Globus Connect Server exposes two interfaces, and a usable filesystem needs both:

The read path subclasses fsspec’s HTTPFileSystem, which already speaks exactly the range dialect GCS serves. The metadata path is a separate Transfer client. Most of the work is in making one object present those two services as a single coherent filesystem.

The 404 that means two things

This is the finding worth carrying away even if you never touch Globus.

GCS load-balances across GridFTP backends. When one is unhealthy it returns an ENDPOINT_ERROR / GCS Manager Internal Error — rendered to the client as HTTP 404. That is the same status a genuinely missing file returns.

Three properties make this nastier than a normal flaky backend:

1. It is indistinguishable by status. Only the response body separates “this file does not exist” from “a backend is sick right now.”
2. It is sticky for the life of a connection. Retrying on the same connection reproduces it. A retry has to establish a fresh one.
3. It is bursty. Observed failure rates on the public test collection swung from 0/20 to 20/20 within minutes, hitting files, directories, and the collection root alike.

The consequence for any client, not just this one:

Three more server quirks

Each of these was found against a live collection, and each forces a design choice:

Suffix ranges return 416. A request for bytes=-8 — which is how parquet readers conventionally seek to the footer — is rejected. Because info() knows the true size from the Transfer API, the workaround is to convert suffix ranges into absolute offsets before they leave the client.

HEAD is unusable. A HEAD 404 carries no body, and the body is the only thing distinguishing a backend fault from a real miss. So HEAD results are permanently ambiguous — there is no amount of retrying that resolves them. Size and existence come from the Transfer API, or from a ranged GET, which does return a body and carries the total in Content-Range.

Mapped vs guest collections need different scopes. A mapped collection requires data_access as a dependent scope of transfer; a guest collection does not, and High Assurance collections must not receive it. The library detects which kind it is talking to rather than asking the caller to know.

Verified against production

Status claims are cheap, so specifically — this has been exercised against three live collections:

• ALCF Eagle (alcf#dtn_eagle, 747 project directories): ls, glob, info, open() with mid-file seek, and sparse ranged reads, all against production Lustre.
• Globus Tutorial Collection 1: writes (PUT/DELETE) round-trip.
• A public collection: anonymous pyarrow column projection — one column of sixty — while that collection was intermittently returning the backend-fault 404s described above. Which is the real test.

ALCF collection UUIDs

ALCF’s documentation lists collection names, not UUIDs, and the API wants UUIDs. Resolved via endpoint_search:

Wrapping up

The interesting part of this project was not the fsspec interface — that is a well-specified surface with good documentation. It was that the transport underneath violates an assumption nearly every HTTP client makes: that a 404 means the thing is not there.

If you are writing anything that talks to Globus over HTTPS, that is the bit to internalize. The rest is plumbing.