marcxmlr

Read, query, analyse and write MARCXML while preserving the structure of complete MARC records.

marcxmlr converts MARCXML to a predictable 11-column canonical table for analysis with R, Arrow and other tabular tools. It can also select complete records before materialisation using a native MARC-aware query engine, or apply the same query afterwards to a tibble or Arrow Dataset. Large catalogues can be streamed to Parquet, with file-level parallel processing for collections distributed across multiple XML files. The canonical data can be validated, edited and written back to MARCXML.

The specialised query interface is optional: it helps with MARC-aware record selection without limiting subsequent analysis using dplyr, data.table, SQL or other tools.

Explore the package:

• Quick start: read and filter a catalogue.
• Canonical representation: understand the 11 columns.
• Working with records in R: analyse, validate and reconstruct MARCXML.
• Selecting complete MARC records: reuse queries before or after materialisation.
• Large collections and Parquet: stream data and use multiple workers.
• Performance evaluation and technical details.

MARC 21 and MARCXML

MARC 21 stands for Machine-Readable Cataloging and is a family of communication formats used to represent and exchange bibliographic, authority, holdings, classification and community information. The examples in this README use bibliographic MARC 21.

MARCXML is the XML representation of MARC 21. It expresses the same logical record structure through XML elements and attributes rather than through the compact traditional MARC serialization.

marcxmlr converts MARCXML into a stable, documented, 11-column canonical representation that preserves the represented MARC structure. Rather than producing a task-specific extraction, it creates a reusable long-form table that can be queried repeatedly as analytical questions evolve.

The package is implemented in R and read_marcxml() returns a tibble, but the canonical data model is independent of R and dplyr. The examples use dplyr for convenience: the same columns can be analysed with data.table and other R tools, or stored in interoperable formats such as Parquet for use with Python, Julia, Rust, SQL engines and other environments.

Installation

Install the current CRAN release with:

install.packages("marcxmlr")

The MARC-aware query interface (marc_has(), marc_filter_records() and filter = during ingestion) requires version 0.5.0 or later. If CRAN still offers an earlier version, install the version from GitHub:

# install.packages("pak")
pak::pak("larry77/marcxmlr")

Source installation from GitHub requires a C toolchain and libxml2 development headers and libraries. On Debian and Ubuntu these are normally provided by r-base-dev and libxml2-dev; Windows source builds use Rtools. Native record prefiltering additionally requires Expat support at package build time (for example, libexpat1-dev on Debian or Ubuntu). Other parsing and writing features remain available on builds without Expat.

Quick start: from MARCXML to complete records

read_marcxml() converts an XML catalogue into a canonical tibble with 11 columns. This is an ordinary R table, but one MARC record occupies multiple rows: its Leader, control fields and data-field subfields are represented separately, tied together by record_id. The structure is explained in the next section.

The bundled GPO sample contains 1,000 bibliographic records. Here we select those containing the subject heading Flood insurance (650$a):

library(marcxmlr)

file <- system.file("extdata", "gpo-cgp-part00-sample.xml", package = "marcxmlr")
q <- marc_has("650", subfield = "a", equals = "Flood insurance")

# Filter the XML before converting the matching records to tabular rows.
selected <- read_marcxml(file, filter = q)
dim(selected)  # 1,320 rows, 11 columns
#> [1] 1320   11
length(unique(selected$record_id))
#> [1] 28

# The very same query works on an already materialised canonical table.
canonical <- read_marcxml(file)
dim(canonical)
#> [1] 53808    11
selected_again <- marc_filter_records(canonical, q)
nrow(selected_again)
#> [1] 1320

Both paths select the same 28 complete records, not just the 650$a rows that matched the query. These are alternatives: filter during ingestion to avoid converting unwanted records, or filter after ingestion if you already have the canonical table. In either case the result is ordinary tabular data, ready for further analysis with dplyr or other tools. The full querying interface, including Boolean combinations and Arrow Datasets, is explained below.

How a MARC record is structured

A MARC record is not a conventional rectangular observation with one value for each variable. It is an ordered structured record. In its traditional machine-readable form, a record contains a Leader, a Directory, and an ordered sequence of variable fields. The Directory is structural bookkeeping for the compact MARC serialization: it records where the variable fields occur in the record.

For analytical purposes, the important content-bearing structure is easier to see by distinguishing three cases:

• a Leader, which occurs once at the beginning of the record;
• control fields such as 001 or 008, which contain a value but no indicators or subfields; and
• data fields such as 100, 245, 264, 650 or 856, which contain a three-digit tag, two indicators, and an ordered sequence of coded subfields.

Both fields and subfields can repeat.

In conventional MARC notation, a title statement might look like:

245 10 $a Nineteen eighty-four / $c George Orwell.

Here 245 is the field tag, 1 and 0 are the two indicators, $a contains the main title information, and $c contains the statement of responsibility. The meaning of a particular tag, indicator or subfield comes from MARC 21; for example, the Library of Congress documents these definitions in the MARC 21 Format for Bibliographic Data.

The same field in MARCXML is:

<datafield tag="245" ind1="1" ind2="0">
  <subfield code="a">Nineteen eighty-four /</subfield>
  <subfield code="c">George Orwell.</subfield>
</datafield>

The XML hierarchy carries information that matters. The two <subfield> elements are children of one particular <datafield> instance, and their order is explicit. Repeated sibling <datafield> elements remain distinct even if they have the same tag and indicators.

A subject field gives another example:

<datafield tag="650" ind1=" " ind2="0">
  <subfield code="a">Totalitarianism</subfield>
  <subfield code="v">Fiction.</subfield>
</datafield>

The blank first indicator is an actual MARC value. It is not missing data. A representation that intends to preserve MARC faithfully has to distinguish that blank from a case where indicators do not apply at all, as for a Leader or control field.

Why a simple flat table is not enough

A tempting analytical representation of a bibliographic record is one row per record:

Such a table may be exactly what one particular analysis needs, but it is not a general representation of the MARC record.

Consider two distinct subject fields:

650 _0 $a Totalitarianism $v Fiction
650 _0 $a Dystopias $v Fiction

These are two different 650 field instances. Each $v Fiction belongs to one particular occurrence of 650.

If values are flattened independently by tag and subfield code, we might obtain:

650$a = "Totalitarianism || Dystopias"
650$v = "Fiction || Fiction"

All values are still present, but field membership has disappeared. With more heterogeneous repeated fields, the original associations cannot in general be reconstructed unambiguously.

The same problem appears inside a single field when a subfield code repeats:

856 40 $u https://example.org/1984 $y Full text $y Mirror

The two $y values are two ordered occurrences inside one specific 856 field. A faithful general representation therefore has to preserve several levels of identity and order:

record
  ↓
field instance
  ↓
field tag, indicators and field position
  ↓
ordered subfield instances
  ↓
subfield code, value, order and occurrence

This is the representational problem addressed by marcxmlr. The package does not force MARC into a task-specific wide table during import. Instead, it first creates a rectangular representation in which the original relationships remain explicit. The analyst can then simplify that representation deliberately when the purpose of the analysis is known.

Design requirements for a general-purpose MARC table

Turning MARCXML into a table is not simply a matter of extracting values. A useful representation must preserve the relationships that give MARC 21 records their meaning while remaining accessible to people and software that work with ordinary tabular data. Four requirements guide the design of marcxmlr:

1. Structural fidelity. Preserve the represented MARC structure and values, including the Leader, control fields, data-field instances, indicators, repeated fields and subfields, and their order. Writing the data back to MARCXML preserves this structure and its values, but not the exact text or formatting of the original XML file; see the discussion of structural preservation.
2. General-purpose tabular interoperability. Use one rectangular table with a fixed set of columns and ordinary scalar values, without list columns, nested objects, specially encoded cells or auxiliary tables. The documented schema is tool-agnostic: it can be analysed using dplyr, data.table, other R tools, or software in Python, Julia, Rust and SQL environments via interoperable tabular formats, including Parquet (subject to each tool’s capabilities and scale limits).
3. MARC-native queryability. Keep tags, indicators and subfield codes directly accessible as columns. Someone familiar with MARC 21 content designation can locate 245$a or 856$u with ordinary table filters, or use marc_has() to select complete records before or after materialisation. See the MARC-aware query interface.
4. Local interpretability. Each row carries enough context to identify its MARC element and, for a subfield, the field instance to which it belongs. Interpreting one row does not depend on joining another table, unpacking a nested value or inferring its meaning from neighbouring rows.

These are design requirements, not a claim of mathematical optimality. A more compact representation might store complete fields in nested cells, distribute the hierarchy across related tables or encode field boundaries through row sequences. Such designs may be useful for particular purposes, but they make ordinary MARC-oriented queries or exchange with generic tabular tools less direct. Compactness alone is therefore not a sufficient measure of quality.

The canonical representation

The canonical representation is marcxmlr’s practical answer to these requirements: a long, fixed-schema table with 11 scalar-valued columns. Each Leader and control field occupies one row; each data-field subfield occupies one row with its field context and structural coordinates. This makes familiar MARC-oriented filtering, grouping and inspection possible with ordinary tabular operations. The 11-column specification and the examples below make this mapping explicit.

The design accepts controlled logical redundancy: field tags, indicators and other field-level information repeat across rows belonging to the same data field. This is deliberate. It allows each subfield to remain intelligible without joins or special decoding, while retaining the relationships needed for reconstruction.

Logical redundancy does not imply equally inefficient physical storage. Parquet supports dictionary and run-length encodings and compression, which can substantially reduce the storage cost of repeated column values. A compressed CSV file (for example, CSV with gzip compression) can also exploit repeated patterns. Actual file sizes depend on the data and the compression settings; no claim is made that this representation is always the smallest. See the Parquet workflow for out-of-memory storage.

The canonical model is neither the only possible lossless tabular mapping nor claimed to be universally optimal. It is a deliberate compromise: alternatives that eliminate redundancy tend to introduce other costs in interpretability, interoperability or MARC-native querying. Improving substantially on this combination of properties is not straightforward.

What one canonical row represents

The row meaning depends on the MARC structure being represented:

• the Leader contributes one row;
• each control field contributes one row; and
• each subfield occurrence of a data field contributes one row.

For a data field with several subfields, field-level information such as the tag, indicators and field position is repeated on every row belonging to that field. That repetition is deliberate. It lets each row carry enough context for direct analysis while still making it possible to identify which rows belong to the same field instance.

MARC tags remain values in the tag column rather than becoming columns of their own. The schema therefore does not change when a catalogue contains an unusual but structurally valid field. MARC knowledge determines what a field means; the canonical representation determines how its structure is retained.

The 11 columns

Every canonical result contains these columns, in this order:

The columns are easier to understand as four related groups.

Identity and structural kind. record_id groups every canonical row produced from the same MARC record. It is deliberately distinct from control field 001, which is source metadata and may be absent, nonnumeric or duplicated across files. field_type distinguishes Leaders, control fields and data fields.

MARC content. tag, subfield_code, value, ind1 and ind2 carry the content designation and values present in the source. For Leaders and control fields, subfield and indicator columns are NA because those concepts do not apply. A blank data-field indicator is instead preserved as the literal one-character string " ".

Structural coordinates. record_id, field_order and subfield_order are the principal coordinates of the hierarchy. Rows from the same data field share the same record_id and field_order; subfield_order preserves the sequence inside that field. These coordinates retain field membership and source order and are the coordinates needed for faithful reconstruction.

Analytical occurrence coordinates. field_occurrence and subfield_occurrence are derived conveniences for analysis. They number repetition directly, making questions such as “the second 650 in this record” or “the second $y in this 856” easy to state. They are not the authoritative source of serialization order; field_order and subfield_order already encode that structure.

The small amount of redundancy is intentional. Repeating field context and storing occurrence numbers makes the table easier to inspect, filter, group and join. The package favours an explicit analyst-friendly representation over a minimally encoded one.

A complete example used throughout this README

The package bundles the following synthetic record as inst/extdata/orwell-marcxml.xml. Every example below refers to this same record.

<record xmlns="http://www.loc.gov/MARC21/slim">
  <leader>00000cam a2200000 i 4500</leader>
  <controlfield tag="001">12345</controlfield>

  <datafield tag="100" ind1="1" ind2=" ">
    <subfield code="a">Orwell, George,</subfield>
    <subfield code="d">1903-1950.</subfield>
  </datafield>

  <datafield tag="245" ind1="1" ind2="0">
    <subfield code="a">Nineteen eighty-four /</subfield>
    <subfield code="c">George Orwell.</subfield>
  </datafield>

  <datafield tag="264" ind1=" " ind2="1">
    <subfield code="a">London :</subfield>
    <subfield code="b">Secker &amp; Warburg,</subfield>
    <subfield code="c">1949.</subfield>
  </datafield>

  <datafield tag="650" ind1=" " ind2="0">
    <subfield code="a">Totalitarianism</subfield>
    <subfield code="v">Fiction.</subfield>
  </datafield>

  <datafield tag="650" ind1=" " ind2="0">
    <subfield code="a">Dystopias.</subfield>
    <subfield code="v">Fiction.</subfield>
  </datafield>

  <datafield tag="856" ind1="4" ind2="0">
    <subfield code="u">https://example.org/1984</subfield>
    <subfield code="y">Full text</subfield>
    <subfield code="y">Mirror</subfield>
  </datafield>
</record>

In canonical form, that one record becomes 16 rows and exactly 11 columns. For readability, <blank> below denotes the actual one-character indicator value " ":

Several features are visible directly:

• both 650 fields have the same tag but different field_order and field_occurrence values;
• the two rows of each 650 share the same field_order, so $a and $v remain attached to the correct field instance;
• the two 856 $y values remain distinct, with subfield_occurrence values 1 and 2;
• indicators remain attached to every row belonging to their data-field instance; and
• source order is retained at both field and subfield level.

Why the representation is analyst-friendly

For someone who knows MARC 21, catalogue questions translate naturally into conditions on documented columns. In plain language:

The package does not need to know that 245 is a title field or that 264 #1 $b identifies a publisher. That semantic knowledge comes from MARC 21. marcxmlr provides a stable representation in which the corresponding conditions can be expressed with ordinary tabular operations.

This is why the representation is deliberately richer than many final analytical products. It preserves relationships before the analyst decides which ones can safely be discarded. Canonical to simplified is easy; simplified to canonical may be impossible.

What semantic preservation means

marcxmlr preserves the represented MARC structure and values, not the incidental byte-level serialization of the XML file. A read/write cycle can preserve Leaders, control fields, data-field instances, indicators, subfields, repetition, values and ordering without reproducing the same indentation, namespace-prefix spelling, attribute order or other formatting details.

This preserve-first principle is the package’s main design choice: field membership, order or repetition discarded during import may be impossible to reconstruct unambiguously afterward.

Working with the same record in R

All examples below use the exact record shown above. It is bundled with the package, so nothing needs to be downloaded.

library(marcxmlr)
library(dplyr)

source_xml <- system.file(
  "extdata",
  "orwell-marcxml.xml",
  package = "marcxmlr"
)

marc <- read_marcxml(source_xml)

dim(marc)
#> [1] 16 11

The result is an ordinary tibble with the canonical columns described above. Now the plain-language conditions can be translated directly into dplyr.

What is the main title?

MARC 21 says that the main title is in 245 $a: tag is 245 and subfield code is a.

marc |>
  filter(tag == "245", subfield_code == "a") |>
  select(record_id, value)
#> # A tibble: 1 × 2
#>   record_id value
#>       <int> <chr>
#> 1         1 Nineteen eighty-four /

Which publisher is named in the publication statement?

For this record, 264 #1 $b identifies the publisher: tag is 264, the second indicator is 1, and the subfield code is b.

marc |>
  filter(tag == "264", ind2 == "1", subfield_code == "b") |>
  select(record_id, value)
#> # A tibble: 1 × 2
#>   record_id value
#>       <int> <chr>
#> 1         1 Secker & Warburg,

The important transition is from a catalogue question, to its MARC content designation, to ordinary table columns. XML traversal is no longer part of the analysis.

Which complete subject field contains 650 $a Totalitarianism?

Sometimes the row that matches is not the whole answer. Here the question is not only to find $a Totalitarianism, but to keep the complete 650 field to which it belongs. Rows from one data-field instance share record_id and field_order, so the field can be selected as a group:

marc |>
  group_by(record_id, field_order) |>
  filter(
    any(
      tag == "650" &
        subfield_code == "a" &
        value == "Totalitarianism"
    )
  ) |>
  ungroup() |>
  select(record_id, field_order, tag, subfield_code, value)
#> # A tibble: 2 × 5
#>   record_id field_order tag   subfield_code value
#>       <int>       <int> <chr> <chr>         <chr>
#> 1         1           5 650   a             Totalitarianism
#> 2         1           5 650   v             Fiction.

The condition matches only $a Totalitarianism, but the result also contains its associated $v Fiction. because both rows belong to the same field.

Inspect repetition directly

The two subject fields remain separate:

marc |>
  filter(tag == "650", subfield_code == "a") |>
  select(record_id, field_order, field_occurrence, value)
#> # A tibble: 2 × 4
#>   record_id field_order field_occurrence value
#>       <int>       <int>            <int> <chr>
#> 1         1           5                1 Totalitarianism
#> 2         1           6                2 Dystopias.

Repetition inside one field is equally explicit:

marc |>
  filter(tag == "856") |>
  select(subfield_code, value, subfield_order, subfield_occurrence)
#> # A tibble: 3 × 4
#>   subfield_code value                    subfield_order subfield_occurrence
#>   <chr>         <chr>                             <int>               <int>
#> 1 u             https://example.org/1984              1                   1
#> 2 y             Full text                             2                   1
#> 3 y             Mirror                                3                   2

Derive a simpler analytical table

The canonical representation preserves structure first. A simpler table can then be derived deliberately for a particular analysis. For example, keep one title and collapse each 650 field to a readable subject string:

titles <- marc |>
  filter(tag == "245", subfield_code == "a") |>
  transmute(record_id, title = value)

subjects <- marc |>
  filter(tag == "650") |>
  group_by(record_id, field_order) |>
  summarise(
    subject = paste(value, collapse = " ; "),
    .groups = "drop"
  ) |>
  group_by(record_id) |>
  summarise(
    subjects = paste(subject, collapse = " | "),
    .groups = "drop"
  )

left_join(titles, subjects, by = "record_id")
#> # A tibble: 1 × 3
#>   record_id title                    subjects
#>       <int> <chr>                    <chr>
#> 1         1 Nineteen eighty-four /   Totalitarianism ; Fiction. | Dystopias. ; Fiction.

This is the intended asymmetry: once the MARC structure is preserved, it is easy to collapse it for a particular purpose. Reconstructing field membership after it has already been discarded may be impossible.

Modify one repeated subfield

Because subfield_occurrence makes repetition explicit, a particular repeated value can be changed without affecting its neighbour. Here only the second 856 $y changes:

edited <- marc |>
  mutate(
    value = if_else(
      record_id == 1L &
        tag == "856" &
        subfield_code == "y" &
        subfield_occurrence == 2L,
      "Backup access",
      value
    )
  )

edited |>
  filter(tag == "856") |>
  select(subfield_code, value, subfield_occurrence)
#> # A tibble: 3 × 3
#>   subfield_code value                    subfield_occurrence
#>   <chr>         <chr>                                  <int>
#> 1 u             https://example.org/1984                   1
#> 2 y             Full text                                  1
#> 3 y             Backup access                              2

Diagnose before writing

After filtering or editing canonical data, diagnose_canonical() can inspect whether the representation still satisfies the structural contract required for safe serialization.

diagnostics <- diagnose_canonical(edited)
diagnostics
#> # A tibble: 0 × 6
#> # ℹ 6 variables: severity <chr>, code <chr>, message <chr>, record_id <int>,
#> #   field_order <int>, subfield_order <int>

An empty diagnostics tibble means that no structural issue was found. The function checks the canonical representation used by marcxmlr; it is not a complete MARC 21 content validator and does not replace MARCXML schema validation.

For Arrow datasets and lazy Arrow queries, diagnose_canonical() validates the representation incrementally without collecting the entire dataset into memory. Records must appear in non-decreasing record_id order. Gaps in record identifiers may produce warnings without indicating structural errors.

write_marcxml() performs the required structural checks itself before writing. Calling diagnose_canonical() explicitly is useful when data have been filtered or edited because the result can be inspected before any XML file is created.

Write MARCXML and read it back

A structurally coherent canonical table can be serialized again:

out_xml <- tempfile(fileext = ".xml")
write_marcxml(edited, out_xml)

reimported <- read_marcxml(out_xml)
identical(edited, reimported)
#> [1] TRUE

Writing the table to MARCXML and reading it again preserves the MARC semantics represented by the canonical table, not incidental XML formatting such as indentation or attribute order.

A complete record can likewise be selected and exported with ordinary table operations before writing.

Handling malformed MARC records

By default, read_marcxml() stops when it encounters a structurally malformed MARC record. Alternatively, on_marc_error = "skip" skips the entire malformed record and continues processing. An error report is required in this mode:

marc <- read_marcxml(
  source_xml,
  on_marc_error = "skip",
  error_report = "marc-errors.csv"
)

The same options are available in marcxml_to_parquet(). The report identifies skipped records and includes the diagnostic reason and original record XML. Source-position gaps in record_id are preserved.

Recovery applies only to MARC-level structural errors. XML syntax and encoding errors remain fatal; the package does not silently repair malformed XML.

Large collections and Parquet

Both the in-memory and Parquet workflows use the same 11-column canonical representation. The practical difference is where that representation lives.

Use read_marcxml() when the source and expanded canonical result fit comfortably in memory. The n_max argument can restrict parsing to the first records, which is useful for previews and development:

preview <- read_marcxml(
  source_xml,
  n_max = 1L
)

unique(preview$record_id)
#> [1] 1

For larger collections, materializing all canonical rows as one R object may be unnecessary. marcxml_to_parquet() writes the same schema to a directory of Parquet files while keeping the expanded canonical data bounded by processing batches.

Arrow support is optional. Install it when needed:

install.packages("arrow")

The bundled example can be converted to Parquet with:

parquet_dir <- tempfile()

marcxml_to_parquet(
  source_xml,
  output_dir = parquet_dir,
  workers = 1L,
  verbose = FALSE
)

catalogue <- arrow::open_dataset(parquet_dir)

catalogue |>
  summarise(rows = n()) |>
  collect()
#> # A tibble: 1 × 1
#>    rows
#>   <int>
#> 1    16

The dataset can be filtered and aggregated with Arrow and dplyr before a smaller result is collected into R:

catalogue |>
  filter(tag == "650", subfield_code == "a") |>
  count(value, sort = TRUE) |>
  collect()

For a catalogue already split across several files:

files <- sort(Sys.glob("data/catalogue/*.xml"))

marcxml_to_parquet(
  files,
  output_dir = "catalogue_parquet",
  workers = 4L
)

When several input files are converted together, record_id values are assigned in deterministic input-file order and remain globally contiguous across the logical dataset.

For one MARCXML file, the optimized sequential path is the natural starting point and additional worker processes may be slower. Parallel execution is more naturally useful when substantial independent files can be assigned to separate workers.

Selecting complete MARC records

MARC records contain many rows in the canonical representation. A conventional dplyr::filter() selects rows; sometimes the desired result is instead every field of each record satisfying a bibliographic condition. The two new public functions address that distinction:

• marc_has() defines a MARC-aware, record-level query; it does not immediately evaluate it.
• marc_filter_records() applies that query to an existing canonical tibble or Arrow Dataset, retaining all rows of selected records.

During XML ingestion, the same query is supplied through the filter argument of read_marcxml() or marcxml_to_parquet(). Native Expat screening identifies matching records, and the native libxml2 reader materialises only those complete records. For restrictive queries this avoids materialising most of the catalogue before deciding what to keep.

Expressing a MARC-aware query

marc_has() supports a field tag (including "LDR" for the Leader), an optional subfield, equals for a case-sensitive exact value, contains for a case-sensitive literal substring, and optional ind1 and ind2. equals and contains cannot be combined in the same call.

# Any record with an electronic location field.
marc_has("856")

# An electronic URL containing a particular domain.
marc_has("856", subfield = "u", contains = "europa.eu")

# A subject field with specified indicators and a controlled-vocabulary code.
marc_has("650", subfield = "2", equals = "EUROVOC",
         ind1 = "0", ind2 = "7")

Queries can be combined with ordinary R operators & (AND), | (OR), and ! (NOT), including nested expressions. The operators work at record level, so different marc_has() conditions may match different fields or subfields of the same record:

q <- (marc_has("650", subfield = "a", contains = "Economics") |
      marc_has("650", subfield = "a", contains = "Trade")) &
     marc_has("856", subfield = "u") &
     !marc_has("999")

Within one marc_has() call, the specified conditions must match the same canonical row. By contrast, two calls combined with & need only match the same record: they are not required to refer to the same occurrence of a repeated data field. Occurrence-specific relationships can be examined using field_order and subfield_order after materialisation.

Combining conditions across MARC fields

Unlike ordinary row filtering, a conjunction can match conditions in different fields of the same record. In the GPO sample, we can select records about flood insurance that also contain a 710 corporate added-entry field:

gpo_xml <- system.file(
  "extdata", "gpo-cgp-part00-sample.xml", package = "marcxmlr"
)

q <- marc_has("650", subfield = "a", equals = "Flood insurance") &
  marc_has("710")

matches <- read_marcxml(gpo_xml, filter = q)
c(records = length(unique(matches$record_id)), rows = nrow(matches))
#> records    rows
#>       9     466

The 650$a and 710 conditions need not occur in the same canonical row; they only need to belong to the same MARC record. All fields of the nine matching records are retained. Queries can also use |, ! and parentheses, as illustrated above.

One identity detail matters when comparing the two execution paths: filtering during ingestion renumbers selected records consecutively, whereas filtering an existing tibble preserves its original record_id values. Selection semantics are the same, but those numeric IDs need not match.

Streaming and Arrow Datasets

The same query can be used to write only selected records to Parquet (requires arrow):

out <- tempfile("gpo-selected-parquet-")
marcxml_to_parquet(
  gpo_xml,
  output_dir = out,
  filter = q,
  batch_records = 200L,
  workers = 1L,
  verbose = FALSE
)

# Query the materialised Parquet Dataset again without collecting it first.
ds <- arrow::open_dataset(out)
records_with_subjects <- ds |>
  marc_filter_records(marc_has("650", subfield = "a")) |>
  dplyr::collect()

The Arrow-backed result stays lazy until collect(); supported filters and joins are translated to Arrow operations. record_id must identify records uniquely across the entire Dataset, including all Parquet fragments.

For multiple input XML files, the same native query can be evaluated independently by file-level workers:

marcxml_to_parquet(
  c(source_xml, gpo_xml),
  output_dir = tempfile("selected-catalogue-"),
  filter = marc_has("856"),
  workers = 2L,
  verbose = FALSE
)

Selected records receive globally unique, consecutive record_id values in input-file order within that conversion. Parallel filtering within one XML file is deliberately unsupported; the supported filtered parallel workflow assigns different input files to different workers.

What remains available outside the query interface

marc_has() is a convenience for selecting complete records, not a general purpose expression evaluator. Arbitrary dplyr predicates cannot run in the native XML prefilter, and marc_filter_records() accepts MARC query objects rather than raw dplyr expressions. After materialisation, however, the canonical table remains an ordinary tibble: users can apply dplyr, regular expressions, custom joins, aggregations or their own analysis without using marc_has() at all. Arrow-backed datasets likewise support the operations that Arrow can translate.

The original filter_tag, filter_subfield and filter_equals arguments are still available for backward compatibility:

read_marcxml(
  gpo_xml,
  filter_tag = "856"
)

Do not combine these legacy arguments with filter = q. Both native filtering interfaces require uncompressed UTF-8 MARCXML without a DTD and an installation built with Expat support. Filtering cannot be combined with MARC-error recovery or chunk_records. Calls without filtering retain the original unfiltered execution paths. If no record matches, the result has the canonical schema with zero rows (or an empty canonical Parquet Dataset).

How the two storage workflows are implemented

The in-memory and Parquet workflows use the same logical representation and, for supported ordinary unfiltered MARCXML, the same package-specific compiled parser. The important difference is how much of the expanded canonical representation is materialized at one time. For filtered XML, Expat first identifies the matching records and their byte offsets; the native reader then materializes those complete records using the same canonical schema.

The optimized sequential unfiltered parser is implemented in C on top of libxml2 and uses two passes over the input. The first pass determines how many canonical rows each selected record will produce. The second pass allocates the required output vectors at their final size and fills them directly while traversing the MARC structure in source order.

For read_marcxml(), the second pass materializes the complete selected result as one tibble. For marcxml_to_parquet(), successive groups of complete records are materialized and written as Parquet parts, so the complete expanded table does not have to exist as one R object. The record remains the natural processing unit because its Leader, fields and subfields must remain structurally coherent.

The same principle applies in the other direction. write_marcxml() can write an in-memory canonical table directly, while Arrow-backed canonical data can be scanned incrementally without first collecting the complete Dataset into one R tibble. Structural relationships are reconstructed from record_id, field_order and subfield_order; the occurrence columns remain analytical coordinates rather than the authoritative source of serialization order.

Readers who want the lower-level details of the libxml2 reader/writer, two-pass allocation, bounded Arrow scans and retained reference implementations can continue with Technical implementation and performance.

Parallel execution in practice

workers = 1L is the default and should normally be the starting point for a single MARCXML file. With one file, requesting multiple workers requires a worker-safe record-based execution path and adds process creation, task coordination and data-transfer overhead. Once the sequential parser itself is fast, those costs can outweigh the available parallel work.

A catalogue already divided into several substantial MARCXML files is a more natural parallel workload. Complete files can be assigned to separate workers, while each worker retains the optimized sequential parser internally. Before multi-file conversion begins, marcxmlr assigns deterministic, non-overlapping record_id ranges from input-file order, so worker completion order does not change the logical record identities in the resulting Dataset.

This is a practical recommendation rather than a universal threshold: CPU, memory, storage, compression, file layout and record complexity all matter. Benchmark representative input before relying on additional workers.

Performance evaluation

The following measurements illustrate the scale of unfiltered reading, Parquet conversion and MARCXML writing. The underlying execution paths remain part of the current package, including version 0.5.0. The measurements were originally collected with marcxmlr 0.3.1 on a Lenovo ThinkPad P50 with an Intel Core i7-6820HQ, 31.18 GiB RAM and a local SSD, using R 4.6.1 and Arrow 25.0.1; they have not been rerun as a 0.5.0 benchmark campaign. Timings indicate scale, not guaranteed performance on other systems.

The single-file parallel measurements are also a useful warning against assuming that more workers are automatically faster: the same 40,000-record Parquet conversion took 19.858 s with two workers and 18.849 s with four, compared with 9.374 s sequentially. Across the 27-file corpus, by contrast, four workers reduced elapsed time from 290.139 s to 118.629 s.

Detailed benchmark methodology, memory measurements and implementation context are provided in Technical implementation and performance.

Scope and further documentation

marcxmlr deliberately separates faithful structural representation from later interpretation. It does not attempt to:

• repair cataloguing records;
• infer missing fields or subfields;
• normalize different cataloguing practices;
• collapse repeated values on behalf of the analyst;
• automatically map a catalogue to another metadata model; or
• replace complete MARC 21 content or MARCXML schema validation.

Its job is to make supported MARCXML available through a stable, structurally faithful canonical table, with optional efficient selection of complete records before or after materialisation. Subsequent reshaping, mapping and analysis remain choices made with general-purpose tools in R or elsewhere.

For further reading:

• Statistical profiling of bibliographic metadata — an applied statistical analysis of 1,000 genuine GPO bibliographic records. It examines metadata coverage, subject indexing, authority identifiers and their relationship with subject-heading complexity.
• Technical implementation and performance — optional deeper reading on compiled parsing, bounded-memory processing, Arrow-backed writing, parallel execution, validation internals and benchmark methodology.

The README and vignette are intended to be sufficient for using the package. The technical note is supplementary material for readers who want to understand how the computational paths are implemented and evaluated.

The implementation principle is the same throughout: preserve the MARC structure first, then simplify or optimize without changing the logical data model.