Raw signal data¶
2428 words Estimated reading time 10 minutes
The raw signal data doesn't contain any information about lightning, only signals with timestamps received by the stations. This data can be used to compute lightning locations by using TOA methods.
Getting the data¶
We provide the raw signal only upon special request for partners and friends in case they want to do some own research. The data will be sent as UDP stream as it comes from the stations without further processing. As the stations have limited capacity the data will be distributed by our servers, but the stations can also send to more than one endpoint. We also provide a JSON encoded UDP data stream upon request.
Data structure¶
The detectors send their data records in plain text to the receiving server. The data records consist of lines that begin with control words followed by further information. There are no blank lines.
Encryption
Starting early 2024 the data can be sent encrypted.
Header data¶
Each data record begins with four lines for the keywords station, processor, time, and position. These four lines are contained in every data sentence, otherwise the data sentence cannot be assigned to the detector.
station <station number>
processor <processor ID>
time <year>-<month>-<day> <hour>:<minute>:<second>[.nanoseconds] [timer]
position <latitude> <longitude> <altitude>
The parameters station, processor, position may not be available due to privacy settings and firmware version! In case of enabled privacy settings you we provide a separate source to assign the correct position.
<station number>is an integer greater than zero.<processor ID>is a unique identifier for the processor and thus a unique identifier for a registered station.<year>-<month>-<day> <hour>:<minute>:<second>corresponds to the point in time at which the signal exceeded a specified threshold value.<nanoseconds>is the number of nanoseconds since the last full second with 9 digits after the decimal point. In case of no GPS signal the nansoseconds (including the dot) will not be available.<timer>is a 64bit counter value derived from the CPU clock (i.e. 84MHz) which will be counted up until reset of the station. This value can be used as a high precision time stamp in case of no GPS signal. This available is available on stations with firmware 10.0 and later. Older firmware versions will not send any signal when there's no GPS signal.<latitude> <longitude> <altitude>: Latitude and longitude are given as a decimal number in degrees. The unit of the altitude is meters.
Additional header data¶
These lines may appear in the header data depending on firmware versions and settings.
<slot_number>is a 16bit number wich will be uniquely assigned randomly to a single station for some hours.<sequence>: A sequence number that is incremented for each signal sent by the station (until reset). Not all signals will be sent to each server.<signal_flags>: bit 1 (lowest) = signal of random noise level; bit 2 = stroke signal; bit 3 = sample of 1pps input; bit 4 = test signal; bit 5 = sample of 5V input; bit 32 = test wave enabled; bit 9 = filtered; bits 10-15 = filter flags. Other bits are reserved for future use.
Examples¶
station 4690
processor 31003C004147313332342455
time 2021-11-13 11:33:48.687089982
position 51.190365 11.942244 169
Footer data¶
Newer firmware versions may send additional information at the end of the data record.
<length>is the length of the whole data record until the end keyword.<tag>is a unique identifier for the data record (internal usage only).
Channel meta data¶
The following five lines for the keywords amp, conversion_time, conversion_gap, values, and start can follow for each channel. These lines do not have to be present in every data record, since this information does not change as often. However, these lines are usually sent every 10 minutes.
amp <channel> <version> <gain> [<filter_frequency>]
conversion_time <channel> <time_nanoseconds>
conversion_gap <channel> <gap_nanoseconds>
values <channel> <number>
start <channel> <offset>
ch <channel> <channel_flags>
<channel>is a number between 0 and 5.<version>is the version of the amplifier channel (i.e.12.2).<gain>describe the gains of the channel. Most channels have two adjustable gains. The value is given as integer. The gain numbers are separated by dots (i.e.16.4)<input>describe the inputs used by the op amp of each channels. Most channels have two op amps with two inputs each. The value is given as integer. The input numbers are separated by dots.<filter_frequency>is the filter frequency in Hz, when a filter is installed and used.<time_nanoseconds>is the time in nanoseconds that is needed for the conversion of a sample (666 by default).<gap_nanoseconds>is the time in nanoseconds between two conversions. It is 2000 by default which corresponds to a sampling rate of 500kHz.<number>is the number of values that are sent for each channel. 512 values are sent by default.<offset>is the offset in the sample which applies to the time stamp. It is 256 by default.<channel_flags>is a bit field which provide the same information as<signal_flags>in the header data, but individually for each channel.
At a sampling rate of 500kHz, a total time range of 1024 microseconds is sent, 512 microseconds before reaching the threshold (= timestamp) and 512 microseconds after passing the threshold.
Example:
Channel signal data¶
<channel>is a number between 0 and 5.<shift>is the shift value for the data values (see below).<coding>is the encoding method for the data values (see below).<bytes>is the number of data bytes that will follow.<binary data>is the data starting with a $ symbol.
Example¶
Data Shifting¶
Each data value is sent as one byte (8 bits) even if more than 8 bits are sampled. For example, if the data is sampled with 12 bits, the shift can be between 0 and 4. If the 4 most significant bits b_11, …, b_8 are zero in all data bytes, then the 8 least significant bits b_7, ...,b_0 are sent with a shift of 0. If this is not the case and the first three most significant bits b_11, b_ 10, b_ 9 are zero in all data bytes, then the 8 bits b_8, ..., b_1 are sent with a shift of 1. If this is not the case and the two most significant bits b_11 and b_10 are zero in all data bytes, then the 8 bits b_9, …, b_2 are sent with a shift of 2. If this is not the case and the first bit b_1 is zero in all data bytes, then the 8 bits b_ 10, …, b_3 are sent with a shift of 3. If this is not the case, then the most significant bit b_11 was not zero in at least one data byte. In this case the 8 most significant bits b_11, …, b_4 are sent with a shift of 4.
Here are some examples for the case that the entire data set in channel 0 consists of only 4 data values.
The keyword coding will be explained later. The key word data is followed by an indication of the number of data bytes that will follow. The third value contains the data starting with a $ symbol.
Example 1¶
Data: <001010101010><000111011101><101110101111><000010101010>
Example 2¶
Data: <001011101010><0001110111010><011110101111><000011101010>
Example 3¶
Data: <000011101010><0000110111010><000000101111><000001101010>
Signal data encoding¶
The signal data can be transmitted as raw data as well as encoded data. Currently only two methods are used for encoding the signal data. Coding 0 means plain binary raw data and coding 3 means encoded data method 3. The two encoded methods 1 and 2 are no longer used. Which encoded method is used is given after the channel number after the key word coding.
The type 3 encoded is a block encoded. The first byte of a block is called the block start byte. The 6 least significant bits of the block start byte represents the block length bl between 1 and 64 (<000000> = 1 and <111111> = 63). The two most significant bits of the block start byte represents the block type bt.
Example¶
The block type 0 indicates a block that has not been encoded. That means, the next bl bytes follow not encoded. The block type 1 indicates a block that consists only of noise. The block start byte is followed by one byte for the maximum noise level and one byte for the minimum noise level. Block type 2 indicates a block with differential encoding. The block start byte is followed by an initial byte.
Perl implementation¶
Below is an example of a decompression implemented in the Perl programming language.
use POSIX;
sub decode_data {
my ($coding, @source) = @_;
my @target;
if ($coding == 0) {
#
# no compression
#
for ($i= 0; $i < (scalar @source); $i++) {
$target[$i]= $source [$i];
}
return ($target);
}
elsif ($coding == 3) {
$source_values= scalar @source;
$source_pos= 1;
$target_pos= 0;
while ($source_pos < (scalar @source)) {
$block_length= ($source[$source_pos]&0x3F)+1;
$block_coding_type= ($source[$source_pos]&0xC0)>>6;
$source_pos++;
if ($block_coding_type == 0) {
#
# raw block
#
while ($block_length > 0) {
@target[$target_pos]= @source[$source_pos];
$target_pos++;
$source_pos++;
$block_length--;
}
}
elsif ($block_coding_type == 1) {
#
# noise block
#
my $max_noise= @source[$source_pos];
$source_pos++;
my $min_noise= @source[$source_pos];
$source_pos++;
my $avg_noise= floor(($max_noise+$min_noise)/2);
while ($block_length > 0)
{
@target[$target_pos]= $avg_noise;
$target_pos++;
$block_length--;
}
}
elsif ($block_coding_type == 2) {
#
# delta block
#
my $first_value= @source[$source_pos];
$source_pos++;
@target[$target_pos]= $first_value;
$target_pos++;
my $last_delta= @source[$source_pos];
if ($last_delta > 127) {
$last_delta-= 256;
}
@target[$target_pos]= $first_value+$last_delta;
$target_pos++;
my $block_num= 0;
while ($block_length > 0) {
my $delta= 0;
if (($block_num%2) == 0) {
$source_pos++;
$delta= (@source[$source_pos]&0xF0) >> 4;
}
else {
$delta= (@source[$source_pos]&0x0F);
}
if (($delta & 0x08) != 0) {
$delta= ($delta&0x07) - 0x08;
}
else {
$delta= ($delta&0x07);
}
$delta+= $last_delta;
@target[$target_pos]= @target[$target_pos-1]+$delta;
$target_pos++;
$last_delta= $delta;
$block_length--;
$block_num++;
}
$source_pos++;
}
}
}
return @target;
}
#
# Test Frame
@source= (0x24,0x6D,0x8A,0x81,0x9F,0x81,0x02,0xD3,0xE0,0x0F,0x3D,0x1F,0xF1,0x12,0x0F,0xF1,0xE1,0x0F,0x3C,0x2D,0x4F,0x2F,0x0E,0x75,0x8A,0x83,0x91,0x8C,0xFE,0x2F,0x1F,0x1F,0x04,0xC3,0xEF,0x11,0xFE,0x21,0xB5,0x86,0x00,0x2E,0x02,0x0F,0xE1,0xF1,0xE2,0xF0,0x11,0x2D,0x1D,0x4E,0xF1,0xF0,0x03,0xF0,0x0F,0xF1,0x0F,0x1F,0x00,0x01,0x10,0x1D,0x2D,0x10,0x11,0xFF,0x6B,0x8B,0x84,0x02,0x8E,0x97,0x9D,0x81,0x9A,0xEE,0xF7,0x85,0x6F,0x12,0x3D,0xCB,0xFC,0x81,0xC3,0xF4,0x07,0x02,0xA9,0xAE,0xB0,0xBF,0xA9,0xF7,0xF4,0x24,0xFE,0xFF,0xB1,0x07,0x05,0xF0,0x31,0x3A,0x1E,0xA1,0x1E,0x0F,0x13,0x4E,0x02,0x1D,0x3D,0x2C,0x00,0xF1,0x1F,0x1F,0x1F,0x3F,0x1F,0x0E,0x3F,0x0E,0x2F,0x21,0x8D,0x81,0x00,0x00,0x10,0xF3,0xFE,0x1F,0xF0,0xF1,0xBF,0x85,0xFF,0x21,0xE2,0xE2,0xE0,0xE2,0xE2,0x0E,0x2F,0x13,0xC3,0x0E,0x2F,0xD2,0xF1,0x10,0x02,0xD1,0x0E,0x1F,0x02,0xE0,0x01,0x03,0xEE,0x2F,0xE4,0xC4,0xD2,0xFF,0x11,0xE3,0xC2,0xBB,0x90,0x01,0xE0,0xF2,0x1E,0x4F,0xFF,0x01,0xE0,0x1F,0x1E,0x12,0x0E,0x1F,0x2C,0x4E,0x1F,0x02,0xE3,0xE1,0xFF,0x01,0x1C,0x4E,0x02,0x0F,0x1F,0x0F,0x11,0xE1,0xFF,0x3D,0x95,0x88,0x01,0x2E,0xD2,0xF0,0x1F,0x10,0x00,0x1F,0x01,0xD3,0xF0,0x0E,0x01,0x87,0x86);
@target= decode_data (3, @source);
print "Source: (" . scalar @source . ") ";
for ($i= 0; $i < (scalar @source); $i++) {
print $source [$i] . " ";
}
print "\n\n";
print "Target: (" . scalar @target . ") ";
for ($i= 0; $i < (scalar @target); $i++) {
print "(" . $i . ")" . $target [$i] . " ";
}
print ".\n\n";
@target= (0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x81,0x83,0x82,0x84,0x84,0x84,0x84,0x83,0x85,0x84,0x84,0x83,0x81,0x80,0x80,0x82,0x84,0x85,0x85,0x86,0x85,0x85,0x85,0x84,0x86,0x84,0x84,0x81,0x82,0x82,0x84,0x85,0x86,0x85,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x8C,0x8A,0x8A,0x89,0x89,0x88,0x88,0x87,0x86,0x89,0x88,0x8A,0x8A,0x89,0x89,0x8A,0x8A,0x88,0x88,0x89,0x86,0x86,0x88,0x88,0x88,0x8A,0x8C,0x8D,0x8C,0x8C,0x8B,0x8B,0x89,0x89,0x88,0x87,0x87,0x88,0x8B,0x8B,0x8C,0x8A,0x8C,0x8C,0x8B,0x8B,0x8A,0x89,0x88,0x8A,0x8B,0x8C,0x8D,0x8D,0x8C,0x8C,0x8C,0x8B,0x8B,0x8A,0x89,0x88,0x87,0x87,0x88,0x89,0x8B,0x8A,0x8B,0x89,0x88,0x87,0x87,0x88,0x88,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x8E,0x97,0x9D,0x9A,0x88,0x75,0x69,0x6F,0x81,0x96,0xA8,0xB6,0xBF,0xC7,0xCB,0xC3,0xB7,0xAB,0xA6,0xA9,0xAE,0xB0,0xA9,0xA0,0x96,0x90,0x8C,0x8C,0x8B,0x88,0x84,0x7F,0x75,0x6C,0x63,0x61,0x5F,0x62,0x64,0x66,0x6B,0x71,0x7A,0x7D,0x81,0x83,0x7F,0x7C,0x7A,0x76,0x72,0x6D,0x69,0x68,0x6B,0x6C,0x6D,0x70,0x74,0x75,0x79,0x7A,0x7D,0x7C,0x7B,0x7A,0x78,0x77,0x77,0x76,0x76,0x75,0x75,0x74,0x76,0x77,0x79,0x7A,0x7B,0x7A,0x7C,0x7D,0x7E,0x7D,0x7E,0x7E,0x80,0x83,0x81,0x81,0x81,0x81,0x82,0x83,0x83,0x86,0x88,0x88,0x89,0x89,0x88,0x87,0x85,0x84,0x85,0x84,0x85,0x87,0x87,0x89,0x89,0x8B,0x8B,0x8B,0x89,0x89,0x87,0x87,0x87,0x85,0x85,0x84,0x84,0x87,0x86,0x88,0x8A,0x8A,0x8C,0x8D,0x8B,0x8B,0x8A,0x8A,0x8B,0x8C,0x8D,0x90,0x90,0x91,0x92,0x91,0x91,0x90,0x8F,0x90,0x8F,0x8E,0x8D,0x8D,0x8D,0x90,0x91,0x90,0x91,0x91,0x8F,0x91,0x8F,0x91,0x90,0x91,0x91,0x90,0x90,0x91,0x90,0x92,0x90,0x90,0x90,0x91,0x90,0x8F,0x8D,0x8D,0x8E,0x8D,0x90,0x92,0x93,0x93,0x93,0x94,0x93,0x92,0x92,0x91,0x91,0x8F,0x8E,0x8F,0x90,0x8F,0x8F,0x8E,0x8F,0x8C,0x8D,0x8C,0x8C,0x8B,0x8A,0x8B,0x8A,0x8C,0x8C,0x8D,0x8D,0x8C,0x8B,0x8B,0x8C,0x89,0x8A,0x89,0x88,0x89,0x8A,0x8A,0x8B,0x8B,0x8B,0x8A,0x8A,0x8B,0x8A,0x8A,0x89,0x87,0x88,0x86,0x88,0x89,0x8C,0x8D,0x8B,0x8B,0x8A,0x89,0x89,0x88,0x88,0x88,0x88,0x88,0x89,0x89,0x89,0x8A,0x88,0x89,0x89,0x89,0x89,0x87,0x87,0x86);
print "Target: (" . scalar @target . ") ";
for ($i= 0; $i < (scalar @target); $i++) {
print "(" . $i . ")" . $target [$i] . " ";
}
print ".\n";