Ditributed dataset profiler

Intermediate
1

Hello, I am working on a data profiler, that profiles data in different formats (MCAP/MATLAB/Rosbag for example) files. Would that be a useful tool? Would like to get some feedback.
Thanks in advance

2

Hmm… trying to build a universal data-quality assessment tool could become an endless challenge​:sweat_smile:, but I can see quite a few practical use cases if the scope is focused:

Yes, I think a cross-format dataset profiler could be useful, especially if it helps people decide what to do with their recorded data without opening a different tool for every format.

For example, I can imagine two workflows where this would help:

  1. Checking a recording before using it. After a robotics experiment, someone wants to know whether the expected camera, IMU, or LiDAR streams were recorded, how much data is available, and whether anything deserves closer inspection before the recording is used for training, evaluation, or debugging.
  2. Inspecting a mixed-format dataset. A research team receives ROS bags, standalone MCAP files, and MATLAB files. A consistent summary of their contents, structure, available time information, and possible anomalies would make the dataset easier to understand and hand off.

There are real adjacent problems here. For instance, this ROS Discourse discussion about rosbag data management mentions dropped frames during recording, dataset management, searching, and reuse. That doesn’t establish demand for any particular profiler, but it illustrates the kinds of tasks a useful one could support.

A practical starting point

If I were approaching this as an exploratory project, I would separate three levels of functionality:

  1. Inventory: What files, topics, channels, variables, message types, shapes, and time ranges are present?
  2. Recording diagnostics: Where are there unusual timestamp gaps, duplicate timestamps, inconsistent rates, or incomplete coverage, when those checks are meaningful?
  3. Application-specific evaluation: Does the recording meet the requirements of a particular task, such as SLAM, perception, or a particular sensor pipeline?

The first level seems useful on its own. The second can be added where timestamps and suitable expectations are available. The third is potentially valuable, but needs application-specific rules rather than one universal definition of a “good dataset.”

I would also keep the observed measurements separate from the decision rules. For example, “maximum recorded gap: 250 ms” is an observation; “this recording is unusable” is a conclusion that needs an expected rate, a tolerance, and some understanding of the recording’s purpose.

What a small report could look like

Something along these lines might already provide value:

Illustrative example only

Asset: run_017/
  Kind: ROS 2 bag
  Storage: MCAP
  Topics: /camera/image, /imu
  Scan scope: indexed summary
  Recorded time range: available
  Detailed gap analysis: not checked

Asset: calibration.mat
  Kind: MATLAB MAT file (v5)
  Variables: K (3x3), bias (3x1)
  Scan scope: variable metadata
  Sample clock: not identified
  Temporal diagnostics: not evaluated

The important part is not the exact output format. It is that the report makes clear what was found, how it was inspected, and what remains unknown.

Two useful references for different parts of this idea are:

  • MCAP CLI, which provides file inspection, summary statistics, and related operations.
  • ROSBag Doctor, which focuses on recording integrity, timing, coverage, and configurable quality checks.

They address different pieces of the problem rather than covering the entire cross-format use case.

If you already have a sample output, even a mock report, that would be interesting to see. It would help distinguish whether your main emphasis is inventory, recording quality, large-scale processing, or something else. The examples above are just possible directions, not assumptions about what you have already implemented.

Possible directions and their trade-offs

1. Cross-format inventory

If the main goal is to make unfamiliar datasets easier to inspect, a unified inventory could be a reasonable first target.

Useful fields might include:

  • File or logical recording identity
  • Format and storage backend
  • File size and available metadata
  • Topics, channels, message types, or variable names
  • Message counts or array shapes, where available
  • Recorded start/end times, when meaningful
  • Reader support and inspection status

The benefit would be a common overview with format-specific details preserved.

For example, a ROS bag’s message count and a MATLAB array’s shape are both useful observations, but they should not be treated as equivalent measurements.

A simple test of value would be whether someone can inspect a handful of mixed-format files and identify which ones deserve closer attention with fewer manual format-specific steps.

2. Recording-quality diagnostics

If the goal is checking recorded data before downstream use, timeline diagnostics become more interesting.

For streams expected to be periodic, one could examine:

  • Inter-message intervals and their distribution
  • Unusually large gaps
  • Repeated or regressing timestamps
  • Start/end coverage
  • Expected topics that are missing
  • Differences between expected and observed rates

However, event-driven topics should not automatically be subjected to fixed-rate expectations. A long interval between alarm or diagnostic messages may be entirely normal.

This is one reason to keep general observations separate from optional policies specifying expected behavior.

ROSBag Doctor provides an example of this distinction: it supports conservative checks without a configuration and more specific thresholds when a recording policy is provided.

3. Application-specific readiness

If the eventual objective is to decide whether a dataset is suitable for a particular application, additional rules become necessary.

For example:

  • A perception workflow might require image streams and matching calibration information.
  • A LiDAR-inertial pipeline might require particular sensor streams, sufficient temporal coverage, and meaningful synchronization checks.
  • A navigation evaluation might need odometry, planning, control, and relevant outcome information.

This is more specialized than inspecting a recording container.

bagx is an interesting reference in this area because it explores task-specific readiness checks rather than relying only on generic file statistics.

One possible architecture would be a format-neutral inventory layer with optional domain-specific checks. That would allow the generic profiler to remain useful without requiring every sensor or application to be understood from the beginning.

4. Comparing recordings and automating decisions

Another possible extension is comparing a recording against an earlier baseline or a declared set of requirements.

Examples include detecting that an expected topic is absent in a new run, or that observed timing characteristics have changed.

A machine-readable report could support comparisons and eventually CI checks, while a human-readable summary would remain useful for exploration.

This does not need to be part of the initial scope. It becomes more attractive once the basic measurements and their meanings are stable.

Some cross-format details worth preserving

There are a few distinctions that seem important for a profiler covering MCAP, MATLAB, and ROS bags.

A logical recording is not necessarily one file

A ROS 2 bag can consist of multiple storage files and associated metadata. It may use SQLite or MCAP as its storage backend.

In particular, rosbag2 has an MCAP storage plugin.

So “ROS bag” and “MCAP” are not necessarily mutually exclusive categories.

It may be useful to distinguish:

  • Logical recording or dataset
  • Physical files belonging to it
  • Storage format or backend
  • Messages, topics, or variables contained within it

This also helps when a recording has been split across multiple files.

MATLAB files do not necessarily define a timeline

A MAT file can simply contain named arrays, structures, and other variables.

For ordinary supported MAT files, SciPy’s whosmat provides variable names, shapes, and classes without requiring the profiler to infer what those variables mean.

A variable might contain timestamps, but the profiler cannot assume that every MAT file has a meaningful sample clock.

MATLAB version support is another useful boundary to state explicitly. SciPy’s loadmat documentation explains that MATLAB v7.3 files require an HDF5-capable reading path rather than SciPy’s ordinary MAT reader.

So a cross-format profiler may benefit from reporting which variant and which reader were actually used.

Metadata inspection and full inspection are different

The MCAP specification makes its summary section optional.

The MCAP implementation notes also explain that some indexed statistics or message indexes may be unavailable depending on how a file was written.

Consequently, absence of summary information does not necessarily mean the file is empty or corrupt.

Depending on the file and reader, obtaining an exact message count or computing detailed timeline statistics may require reading the actual messages.

A practical report could distinguish operations such as:

metadata_only
indexed_summary
sampled
full_scan

It could also distinguish whether a reported count is exact, estimated, or unavailable.

This would let users choose between inexpensive inspection and deeper analysis without mistaking the former for a complete verification.

Unknown is not the same as zero

This distinction seems especially valuable in a multi-format report.

For example:

Status Meaning
observed The relevant value was actually inspected or measured
not_checked The operation was not performed
not_available The required information was unavailable through the chosen reading path
not_applicable The check does not meaningfully apply to this data
reader_error Inspection was attempted but failed

These labels are only suggestions, not a proposed universal standard.

For instance, a dataset with no defined time axis should not automatically receive “0 Hz” or “no timing problems.” Similarly, an unavailable checksum is not evidence that an integrity check passed.

Recording the inspection method and reader version alongside the result could also make comparisons more reproducible.

Timing checks and what they can actually establish

Time-related measurements are useful, but their interpretation depends on the clock and reading order.

1. Different timestamps can describe different events

The MCAP format distinguishes log_time and publish_time.

A serialized message may also contain its own sensor-origin timestamp. These time values need not be identical.

ROS 2 recording behavior can also involve different received and published timestamps, depending on the recorder, storage path, and software version.

For a profiler, I would therefore consider recording the timestamp basis used for each calculation.

For example:

clock_basis: recorded_log_time
order_basis: original_file_order
metric: maximum_positive_interval
scan_scope: full_scan

This helps prevent a number that was computed correctly from being interpreted as measuring something else.

2. Sorting can change which anomalies are visible

The MCAP Python reader API explicitly supports reading messages in log-time order or in file order.

Suppose messages appear in a file with timestamps:

1.0, 1.1, 1.3, 1.2, 1.4

Inspection in the original order reveals a timestamp regression.

If the reader sorts by timestamp first, the resulting sequence is monotonic. That is useful for some interval analyses, but it no longer answers whether the timestamps were monotonic in the original ordering.

Neither reading mode is inherently wrong. They answer different questions.

If the purpose is to detect ordering irregularities, retaining the relevant original order matters. If the purpose is temporal coverage or an ordered timeline, sorting may be appropriate.

3. A recorded gap is not automatically a dropped sensor frame

Suppose a nominally periodic stream contains an unusually large interval.

That might reflect missing messages, delayed publication, recording-side loss, a clock discontinuity, or another mechanism.

The recorded timestamps alone generally do not establish which explanation is correct.

For a periodic sensor with a known expected cadence, the observation could be reported as an unusual gap. For an event-driven topic, the same interval may be expected.

A useful minimal control would be to compare:

  • A known regular sequence
  • The same sequence with an intentionally introduced gap
  • An event-driven sequence where irregular intervals are normal

This tests whether the diagnostic distinguishes the observation from its interpretation.

4. Synchronization requires additional assumptions

Comparing timestamps between two topics can reveal useful differences, but it does not automatically prove that the physical sensors were synchronized.

The result depends on which timestamps are being compared, clock relationships, message semantics, and any required calibration or synchronization assumptions.

A general profiler could report recorded timestamp offsets while leaving stronger application-specific synchronization judgments to optional checks.

Again, I think the useful division is:

What was measured? Under which assumptions? What decision rule, if any, was applied?

Large datasets and distributed profiling

If “distributed” in the title refers to parallel or distributed execution, there is another interesting design branch.

The best approach depends on what is expensive.

Many independent files

When profiling many independent files, per-file parallel processing may be enough to obtain useful scaling.

Each worker can produce a bounded, structured summary, with results combined at the dataset level.

This also creates a natural place for retries, incremental processing, and caching results for unchanged files.

One very large recording

For a single large file, the situation is different.

Indexed metadata access, selective reading, sequential scans, decompression, and payload decoding can have very different costs.

For MCAP, indexes can make some operations inexpensive when the relevant data structures are present. However, a full scan may still be required for detailed statistics.

The MCAP CLI documentation discusses indexed reads and remote-file access, including cases that require full remote scans.

There is also a reader-level detail worth considering: the MCAP Python reader documentation warns that sorting a non-seekable input by log time can require loading the content into memory.

That could matter when designing a bounded-memory profiler for large or remote recordings.

Statistics across partitions

Simple counts, minima, and maxima are often straightforward to combine, assuming consistent definitions and complete input coverage.

Timeline statistics can be less straightforward.

Consider this purely illustrative example:

Partition A: 0.0, 0.1, 0.2 seconds
Partition B: 0.8, 0.9, 1.0 seconds

Maximum internal gap in A: 0.1 seconds
Maximum internal gap in B: 0.1 seconds

Gap between partitions:   0.6 seconds

Combining only the two local maximum-gap values would miss the larger gap between the partitions.

For ordered, non-overlapping partitions, a worker could preserve boundary information such as:

first_timestamp
last_timestamp
message_count
internal_max_gap
clock_basis

The aggregator could then evaluate the boundary between adjacent partitions as well.

But this simple method depends on the partitions really being ordered and non-overlapping. Overlapping intervals, interleaved streams, duplicate timestamps, or clock resets require more careful handling.

Likewise, exact percentiles, jitter statistics, and some synchronization measurements cannot generally be obtained by naively averaging worker-level results.

What I would measure before adding more infrastructure

If large-scale processing is the primary goal, useful performance measurements would include:

  • Metadata-only versus full-scan time
  • Bytes read and remote requests
  • Peak memory consumption
  • Decompression or payload-decoding cost
  • Retry and partial-failure behavior
  • Whether processing multiple files independently already provides sufficient throughput

These measurements would help distinguish a reader/I/O bottleneck from a problem that actually benefits from distributed computation.

None of this implies that a distributed framework is necessary. It is just one direction if dataset size or workload eventually makes it worthwhile.

Small ways to validate usefulness and further references

I think the most informative early test would be an end-to-end task rather than a broad feature checklist.

For example:

Test A: Mixed-format inspection

Take two or three small files with different representations. Can a user obtain a consistent overview and decide what to inspect next without manually switching among specialized tools?

A useful output is not necessarily a quality score. It might simply identify an expected stream, variable, or time range that requires further attention.

Test B: Known-good versus deliberately altered recording

For a regular stream, introduce a known timing irregularity or duplicate timestamp.

Check whether the profiler identifies the observation and describes it without attributing a cause it cannot establish.

An event-driven stream can serve as a control against overly broad rate or gap warnings.

Test C: Fast summary versus full scan

Compare an indexed MCAP file and a valid one without a summary section.

The useful question is whether the report accurately distinguishes information available from metadata from measurements requiring a full scan.

Test D: Partition boundaries, if relevant

If distributed timeline processing is planned, compare a whole-recording analysis with partitioned analysis on a small synthetic sequence containing a known boundary gap.

Agreement under defined assumptions would be a useful starting point before large benchmarks.

These are possible validation paths, not prerequisites for establishing whether the project is worth exploring.

A few related implementation references, depending on the direction:

  • Rosbags: ROS bag reading and storage handling in Python.
  • bagx: application-specific recording readiness and structured findings.
  • Rosbag Resurrector: broader recording inspection and processing workflows.
  • SciPy MAT-file utilities: MATLAB file inspection and reading, with format-dependent limitations.

They are useful examples of different layers or workflows rather than direct replacements for a general cross-format profiler.

For me, the most convincing success criterion would be simple:

Can the profiler help someone make a useful decision about unfamiliar recorded data, with less manual work, while making clear which conclusions are supported and which are still unknown?

If it can do that reliably for even a focused group of formats and use cases, I think that would already be a meaningful tool.

3

Hey John6666,

Really appreciate you for taking the time to give a detailed response.:folded_hands: . I will take your suggestion and implement the #1 in practical starting point (just the inventory profile) and take it from there.

Thanks again