Import and export pipeline

This page covers how files get into and out of a map: the editor’s import wizard, the cookie-authenticated map-io routes, export, and the paging helpers every large read uses. The public API’s upsert import is on Public API internals.

Where the code is: apps/app/src/server/functions/data-upload.ts, server/lib/map-io/, server/lib/pagination.ts, lib/csv-conventions.ts, routes/api/maps/$mapId/.

The editor’s import flow

The Import / Export dialog (in the map’s header) drives CSV and GraphML through one pipeline: upload, scan, map, ingest. All functions use orgMiddleware.

1. getSignedUploadUrl({ map_id, filename, mime_type }) asserts map access, resolves the org from map_id server-side, and returns a signed upload token for the private map-imports bucket at {orgId}/{mapId}/{uuid}-{filename}. Minting the token needs the service-role client. The browser uploads straight to Storage; the app server never handles the bytes.

2. createImportSession({ map_id, format, files }) takes one or two { filename, storage_path, role? } entries. role is elements or connections, set by the wizard and not re-derived. A file with no role counts as elements. It refuses with IMPORT_FILE_ROLES_INVALID when there are no files, more than two, a GraphML import with more than one, two files of the same role, or a connections file with no elements file. It inserts an app.import_session row (status: 'scanning') with the RLS client, downloads the files with the service-role client (signed-token uploads aren’t readable through RLS), and scans them with analyzeCsvBundle or analyzeGraphmlAttributes. Both receive the map’s existing fields (loadCandidateFields) so they can suggest linking to one. The scan goes into scan, the suggestion into column_mapping, and the status moves to previewing. A read or parse failure sets invalid with an error and throws IMPORT_SCAN_FAILED.

3. updateImportMapping({ session_id, column_mapping }) validates and saves the user’s mapping, then sets mapping. For CSV it refuses (IMPORT_MAPPING_INVALID):

   • more than one label, or no label and not exactly one id;
   • a connections file without exactly one from and one to;
   • more than one connection_value, or one with no connection_type_values column;
   • more than one of each date role;
   • any role the scan ruled out for that column (assertRolesAllowed).

   Two columns that would create a same-named field of the same subject, or an empty name, fail with IMPORT_MAPPING_NAME_CONFLICT. GraphML has no hard requirement.

4. ingestImportSession({ session_id }) asserts map access and that the session is previewing or mapping (else IMPORT_SESSION_NOT_READY). CSV needs a label or id column (IMPORT_MAPPING_INCOMPLETE). A CSV bundle with a connections file, and any GraphML import, must have reached mapping, so a field-creating mapping is never ingested straight off the suggestion. It sets ingesting, detaches runIngestTask without awaiting it, and returns { started: true }.

5. runIngestTask runs after the request ends, so it has no cookies. It uses the service-role client, downloads the files again, calls ingestCsvBundle or ingestGraph with the saved mapping, records one map.imported change for the session’s created_by (see Editing internals), and writes ingested plus result, or failed plus error.

6. The client polls getImportStatus (a POST, because repeated identical GET server functions fail intermittently). getImportPreview re-reads a session. Both are RLS-scoped reads; import_session’s policy is the access check.

removeImportRow exists but throws NOT_IMPLEMENTED.

Status lifecycle (app.import_status):

scanning → previewing | invalid
previewing → mapping → ingesting → ingested | failed

A session left in ingesting by a server restart is marked failed by the maintenance sweep (failStuckImportSessions in server/lib/maintenance.ts) once it is more than six hours old. The sweep runs from POST /api/maintenance/purge.

CSV analysis

CSV follows Kumu’s spreadsheet conventions: an elements file (Label or ID first) and an optional connections file (From, then To), | between values, and Direction one of directed, undirected, mutual. The user-facing format is on Import format.

lib/csv-conventions.ts holds the header rules, detectCsvFileRole, splitMultiValue and parseDirection. It has no imports because both a React component and map-io/csv.ts use it; importing from server/ would pull server code into the browser bundle.

analyzeCsvBundle (map-io/csv.ts) profiles every column, works out the roles each column may take, and suggests a mapping. It writes nothing.

Profiling is sampled. profileColumn reads PROFILE_SAMPLE_ROWS (200) rows, strided across the file so a sorted export isn’t misread. The profile records valueType (numeric, boolean or null for text), hasMultiValue, isEmpty, looksLikeDirection, looksLikeDate and a few sample values. A profile can be wrong about rows it didn’t see; ingest drops a mismatched cell for that row only.

allowedRolesFor(profile, fileRole) is the authority on roles. The mapping UI disables everything else, and updateImportMapping re-checks it. CSV_ROLES_BY_FILE offers each file only its own roles.

Role	File	Allowed when	Creates
label, id, description	both	single-valued	an elements label is the node label (an id stands in when there’s no label); everything else becomes a property
from, to	connections	not empty	endpoints
direction	connections	single-valued, every cell a direction token	property direction (see Direction)
measure	elements	single-valued, numeric	one network.measure
tag	elements	single-valued, boolean	one ungrouped network.tag
tag_values	elements	text	one tag per distinct value
connection_type	connections	single-valued, numeric or boolean	one network.connection_type
connection_type_values	connections	text	one connection type per distinct value
connection_value	connections	single-valued, numeric	nothing; the value on each row’s typed connection
connection_date, connection_start, connection_end	connections	single-valued, every cell a partial date	the connection’s occurred_on, valid_from, valid_to
property, ignore	both	always	a property, or nothing

Suggestions:

• An elements Type or Tags column, and a connections Type or Tags column, default to the per-value role.
• Date roles are suggested by header only (Date, Year, Start date, Since, Until and so on), never by values, since four-digit amounts look like years.
• With a Type column present, the first numeric column is suggested as connection_value.
• Any other numeric or boolean column becomes a single-field role, linked to an existing field of the same name and kind if there is one. Free text becomes a property. An empty column is ignored.
• There is no strength role. Kumu’s Strength is an ordinary numeric column.

Per-value roles don’t list their values in the mapping. Ingest collects the distinct values over the whole file (distinctValuesFor). Past MAX_VALUE_FIELDS (24) values the column becomes a single property, with a warning. Each value reuses an existing row of that name and kind, or creates one. A connection type is reused only if it is boolean, unless a connection_value column is present: then an existing boolean type is switched to record a value through network.set_connection_type_value_type.

An entry’s optional name renames what the column creates. undefined means the header.

Legacy mappings. normalizeCsvMapping reads a stored mapping with no elements key as a nodes-only mapping, and upgrades the old field and values roles. upgradeAllowedRoles does the same for a stored scan’s allowedRoles.

Endpoint resolution

readElementKeys → buildEndpointIndex → resolveEndpoint in map-io/csv.ts. The scan’s pre-flight and ingest use the same functions, so the warnings match what ingest does.

Each From/To key resolves in this order:

1. element ID;
2. exact label;
3. case-insensitive label.

A repeated ID drops the second row (element_duplicate_id). A repeated label is allowed but makes that label ambiguous: an exact unique string still resolves, and only the case-insensitive fallback fails (connection_ambiguous_endpoint). The pre-flight reports unresolved, ambiguous, self-connection and blank-endpoint counts with example rows in CsvBundleScan.bundleWarnings.

Only endpoints in the same upload resolve. A connections file can’t attach to nodes already in the map.

CSV ingest

ingestCsvBundle (map-io/csv-ingest.ts) plans every catalogue row for both files with ids minted in planning, creates them in one insertCatalogueRows call, then writes in two phases.

• Phase A inserts the elements as unplaced network.node rows (no pos_x/pos_y) in chunks of INSERT_CHUNK_SIZE (500), then their measures and tags.
• Phase B runs only with a connections file. It resolves endpoints and inserts connections.

Rules:

• From and To fan out over the product of both cells split on |.
• Self-connections are skipped before insert (connection_self_loop).
• Each field binding on a row is its own network.connection row, sharing endpoints and properties. A row with no binding gets one placeholder with metric_id: null.
• connection_value puts the row’s number on its connection_type_values connection, keeping one row. A blank or non-numeric value is no value.
• A cell that doesn’t fit its column’s kind is dropped for that row only. Tags and boolean types store only true values.
• Element rows with no label count in rowsSkipped. Dropped endpoint pairs count in connectionRowsSkipped and are itemised in issues with example rows.

Tag groups come from the data. A single-valued tag_values column creates one network.tag_group named after the column. A column with any pipe-separated cell stays ungrouped, because a node can hold only one tag per group. A connection_type_values column uses its name as the types’ category.

Existing grouped tags keep their group. ingestCsvBundle and ingestGraph read each tag’s group from network.tag before writing node_tag (loadTagCatalogue, stampTagMemberships in ingest.ts). If one node would get two tags from one group, the first in file order is kept and the rest are dropped with one warning.

Imports are not transactional. A failure part-way leaves what was already written. That is why ingest drops conflicting rows rather than letting a constraint reject a batch.

An element ID is kept as a property, not as an identity. Re-importing creates duplicates. A connection ID is likewise a property.

Direction

Direction belongs to the connection type (network.connection_type.directionality), not to each connection. Kumu records it per row. So:

1. A type created from a column, or from one value of a per-value column, takes the direction its rows agree on (deriveDirectionality). Mixed or absent directions give undirected, with a warning when mixed.
2. A linked existing type is never changed. A disagreement is reported in IngestResult.warnings.
3. The raw cell is always stored in connection.properties.direction.
4. From/To are never swapped, and mutual is never faked by duplicating rows.

GraphML

analyzeGraphmlAttributes (map-io/graphml.ts) groups every non-reserved attribute by name and subject, and suggests one resolution per key:

Resolution	Meaning
existing	bind to a field by field:{id} key or by name and subject
new	create a numeric or boolean field (optional name)
property	store as a node or connection property
ignore	drop
label	use as the node label

A node attribute named like label, name or title is suggested as label when no node in the file has an explicit label.

Per-value explosion. A connection attribute named type, edgetype, kind or category, with 2 to 12 distinct text values (MAX_CATEGORICAL_VALUES), becomes one mapping row per value (attrValueKeyOf). Each value defaults to a new boolean field, or links to an existing field of that name. Its direction is directed only if every edge carrying it is directed.

ingestGraph (map-io/ingest.ts) uses only the confirmed mapping. Each attribute bound to a field becomes its own connection row; the rest of the edge’s attributes become properties and are listed in unmatchedFieldKeys. Nodes arrive unplaced.

map-io server routes

Two TanStack Start server routes under routes/api/maps/$mapId/. Function middleware can’t attach to server routes, so each handler checks supabase.auth.getClaims() (401) and assertMapAccess (403) itself, then uses the RLS-scoped client. The orchestration lives in server/lib/map-io/service.ts, shared with the public API.

Route	Formats	Notes
GET /api/maps/$mapId/export?format=	graphml, csv, json	Used by the map’s download links and Duplicate map
POST /api/maps/$mapId/import?format=	graphml, json	Body is the file; 413 over 10 MB by Content-Length, checked before reading

Import through this route is create-only. Every node and connection in the file is new; nothing is matched or updated. Each import records one map.imported change for the caller.

• GraphML uses programmaticAttributeMapping: the wizard’s suggestion with every non-existing resolution turned into property, so the route never creates a field.
• Internal JSON adopts the file’s own fields[] (adoptInternalJsonFields). A field whose id is already in the map as the same kind is reused. Any other is created under a new id, with tags regrouped by category, and the document’s references are rewritten. So a map’s JSON export imports into any map. A fieldId the file uses but doesn’t describe fails the import. False tag and boolean values are dropped.
• The JSON response includes nodeIds (file node id → new node id). Duplicate map (components/home/duplicate-map.ts) exports the source as JSON, imports it into the new map, and places each copy with placementsFor and those ids.

Export

loadGraphForExport (map-io/load.ts) reads the map’s nodes, connections and its whole catalogue (readMapFields, unused definitions included). renderExport serialises it.

• GraphML (buildGraphml): field keys are field:{id} and property keys prop:{key}, so they never collide with each other or with label, and a re-import binds by id. Connections with no type are left out.
• CSV (buildCsv): nodes only, with label, every property key and every node field by name. It does not round-trip a graph.
• Internal JSON (buildInternalJson): everything, including untyped connections.

Paged reads

PostgREST caps an unranged query at max_rows (1000 in supabase/config.toml) with no error. Any read that can pass that must page through server/lib/pagination.ts.

fetchAllRows(errorContext, buildQuery, { orderBy }) appends .order(orderBy).range(from, to) to each 1000-row page itself. Offset paging needs a total order, or pages overlap or leave gaps. orderBy defaults to id and must be unique. A caller wanting its own sort adds .order() in the builder, and orderBy becomes the tiebreaker. Pass a builder function, because supabase-js builders mutate.

fetchAllRowsIn(errorContext, ids, buildQuery) is for a filter over a list of ids. supabase-js puts .in() lists in the URL, and the gateway rejects URLs over about 15 KB (roughly 390 UUIDs). It de-duplicates and sorts the ids, splits them into chunks of 150 (IN_FILTER_CHUNK_SIZE), pages each chunk with up to four running at once, and concatenates the results. When the .in() column is the orderBy column the result is in overall id order. An empty list returns [] without a request.

Whole-map reads filter by map_id and need no id list.