File Generation¶
The telematics gateway continuously records machine data in log files (.clf files) and transmits them to the DataPlatform.
The DataPump Service processes these files to generate the export files delivered to the customer's cloud storage.
Time-series data and DTC data follow different file generation rules and are generated independently — there is no direct file-to-file correspondence between CSV and JSON files.
To correlate DTC and time-series data, match the timestamp field in the CSV against the start and end fields in the DTC JSON files.
Time-Series Files¶
Time-series data is exported as zipped .csv files. A new file is created either when the configured time interval expires or when sufficient data has accumulated — whichever comes first.
A single CSV file aggregates data from up to 1000 .clf files.
The file generation interval can be configured individually for each DataPump setup by Proemion Customer Success. The configurable interval range is 30–300 seconds. The default interval is 150 seconds.
Note
Smaller intervals increase update frequency but result in disproportionately higher transmission and storage costs.
The default values represent a compromise between update frequency and file size efficiency.
File sizes depend on the number of machines, signals, and data points in the export. The following are estimates for a setup with 1000 machines and a full batch of 1000 .clf files. Actual sizes will be smaller for smaller batches or fewer machines.
| Uncompressed CSV | gzip compressed | |
|---|---|---|
| Expected (median) | ~70 MB | ~5–9 MB |
| Upper bound (~p98) | ~100 MB | ~10–13 MB |
For the full file format specification including field definitions and examples, see CSV File Structure.
DTC Files¶
DTCs are treated as events with state transitions and lamp states, and are exported as .json files.
One .json file is generated per .clf file received. DTC files are delivered without batching — data is typically available in the customer's cloud storage within 5 seconds of receipt on the DataPlatform.
For the full DTC file format specification including field definitions, transmission behavior, and examples, see DTC File Structure.
Data Retention and Delivery¶
If the customer's cloud storage is temporarily unavailable, the DataPlatform retains data in a persistent queue and continues retrying delivery. Data is retained for up to 24 hours. If a file delivery is interrupted, the file is re-delivered automatically.