Raw signal data¶
1965 words Estimated reading time 8 minutes
The raw signal data doesn't contain any information about lightning, only signals with timestamps received by the stations. This data can be used to compute lightning locations by using TOA methods.
Getting the data¶
We provide the raw signal only upon special request for partners and friends in case they want to do some own research. The data will be sent as UDP stream as it comes from the stations without further processing. We also provide a JSON encoded data stream upon request.
Data structure¶
The detectors send their data records in plain text to the receiving server.
Encryption
Starting early 2024 the data can be sent encrypted.
The data records consist of lines that begin with control words followed by further information. Each data record begins with four lines for the keywords station, processor, time, and position. These four lines are contained in every data sentence, otherwise the data sentence cannot be assigned to the detector.
station <station number>
processor <processor ID>
time <year>-<month>-<day> <hour>:<minute>:<second>
position <latitude> <longitude> <altitude>
Example:
station 4690
processor 31003C004147313332342455
time 2021-11-13 11:33:48.687089982
position 51.190365 11.942244 169
<station number> is an integer greater than zero. <processor ID> is a unique identifier for the processor and thus a unique identifier for a registered station. The time stamp corresponds to the point in time at which the signal exceeded a specified threshold value. The seconds in the time line are specified with 9 digits after the decimal point, i.e. in nanoseconds. Latitude and longitude are given as a decimal number in degrees, not in degrees, minutes, seconds, etc. The unit of the altitude is meters.
The following five lines for the keywords amp, conversion_time, conversion_gap, values, and start can follow for each channel. These lines do not have to be present in every data record, since this information does not change as often. However, these lines are usually sent every 10 minutes.
amp <channel> <firmware> <board> 0 0
conversion_time <channel> <microseconds>
conversion_gap <channel> <microseconds>
values <channel> <number>
start <channel> <number>
Example:
<channel> is a number between 0 and 5. The sampling time for a 12-bit sample is usually 666 nanoseconds, the difference between two start points of valuations, the conversion gap, should be 2000 nanoseconds. Other values are not accepted here. This setting corresponds to a sampling rate of 500kHz. 512 values should be sampled in total, 256 values before reaching the threshold and 256 values after passing the threshold. At a sampling rate of 500kHz, a total time range of 1024 microseconds is sent, 512 microseconds before reaching the threshold and 512 microseconds after passing the threshold.
Example:
Each data value is sent as one byte (8 bits) even if more than 8 bits are sampled. For example, if the data is sampled with 12 bits, the shift can be between 0 and 4. If the 4 most significant bits b_11, …, b_8 are zero in all data bytes, then the 8 least significant bits b_7, ...,b_0 are sent with a shift of 0. If this is not the case and the first three most significant bits b_11, b_ 10, b_ 9 are zero in all data bytes, then the 8 bits b_8, ..., b_1 are sent with a shift of 1. If this is not the case and the two most significant bits b_11 and b_10 are zero in all data bytes, then the 8 bits b_9, …, b_2 are sent with a shift of 2. If this is not the case and the first bit b_1 is zero in all data bytes, then the 8 bits b_ 10, …, b_3 are sent with a shift of 3. If this is not the case, then the most significant bit b_11 was not zero in at least one data byte. In this case the 8 most significant bits b_11, …, b_4 are sent with a shift of 4.
Here are some examples for the case that the entire data set in channel 0 consists of only 4 data values.
The keyword coding will be explained later. The key word data is followed by an indication of the number of data bytes that will follow. The third value contains the data starting with a $ symbol.
Example 1.
Data: <001010101010><000111011101><101110101111><000010101010>
Example 2.
Data: <001011101010><0001110111010><011110101111><000011101010>
Example 3.
Data: <000011101010><0000110111010><000000101111><000001101010>
Signal data encoding¶
The signal data can be transmitted as raw data as well as encoded data. Currently only two methods are used for encoding the signal data. Coding 0 means plain binary raw data and coding 3 means encoded data method 3. The two encoded methods 1 and 2 are no longer used. Which encoded method is used is given after the channel number after the key word coding.
The type 3 encoded is a block encoded. The first byte of a block is called the block start byte. The 6 least significant bits of the block start byte represents the block length bl between 1 and 64 (<000000> = 1 and <111111> = 63). The two most significant bits of the block start byte represents the block type bt.
Example:
<10001010>
bt = <10> = 2
bl = <001010> = 21
The block type 0 indicates a block that has not been encoded. That means, the next bl bytes follow not encoded. The block type 1 indicates a block that consists only of noise. The block start byte is followed by one byte for the maximum noise level and one byte for the minimum noise level. Block type 2 indicates a block with differential encoding. The block start byte is followed by an initial byte.
Example decoding¶
Below is an example of a decompression implemented in the Perl programming language.
use POSIX;
sub decode_data {
my ($coding, @source) = @_;
my @target;
if ($coding == 0) {
#
# no compression
#
for ($i= 0; $i < (scalar @source); $i++) {
$target[$i]= $source [$i];
}
return ($target);
}
elsif ($coding == 3) {
$source_values= scalar @source;
$source_pos= 1;
$target_pos= 0;
while ($source_pos < (scalar @source)) {
$block_length= ($source[$source_pos]&0x3F)+1;
$block_coding_type= ($source[$source_pos]&0xC0)>>6;
$source_pos++;
if ($block_coding_type == 0) {
#
# raw block
#
while ($block_length > 0) {
@target[$target_pos]= @source[$source_pos];
$target_pos++;
$source_pos++;
$block_length--;
}
}
elsif ($block_coding_type == 1) {
#
# noise block
#
my $max_noise= @source[$source_pos];
$source_pos++;
my $min_noise= @source[$source_pos];
$source_pos++;
my $avg_noise= floor(($max_noise+$min_noise)/2);
while ($block_length > 0)
{
@target[$target_pos]= $avg_noise;
$target_pos++;
$block_length--;
}
}
elsif ($block_coding_type == 2) {
#
# delta block
#
my $first_value= @source[$source_pos];
$source_pos++;
@target[$target_pos]= $first_value;
$target_pos++;
my $last_delta= @source[$source_pos];
if ($last_delta > 127) {
$last_delta-= 256;
}
@target[$target_pos]= $first_value+$last_delta;
$target_pos++;
my $block_num= 0;
while ($block_length > 0) {
my $delta= 0;
if (($block_num%2) == 0) {
$source_pos++;
$delta= (@source[$source_pos]&0xF0) >> 4;
}
else {
$delta= (@source[$source_pos]&0x0F);
}
if (($delta & 0x08) != 0) {
$delta= ($delta&0x07) - 0x08;
}
else {
$delta= ($delta&0x07);
}
$delta+= $last_delta;
@target[$target_pos]= @target[$target_pos-1]+$delta;
$target_pos++;
$last_delta= $delta;
$block_length--;
$block_num++;
}
$source_pos++;
}
}
}
return @target;
}
#
# Test Frame
@source= (0x24,0x6D,0x8A,0x81,0x9F,0x81,0x02,0xD3,0xE0,0x0F,0x3D,0x1F,0xF1,0x12,0x0F,0xF1,0xE1,0x0F,0x3C,0x2D,0x4F,0x2F,0x0E,0x75,0x8A,0x83,0x91,0x8C,0xFE,0x2F,0x1F,0x1F,0x04,0xC3,0xEF,0x11,0xFE,0x21,0xB5,0x86,0x00,0x2E,0x02,0x0F,0xE1,0xF1,0xE2,0xF0,0x11,0x2D,0x1D,0x4E,0xF1,0xF0,0x03,0xF0,0x0F,0xF1,0x0F,0x1F,0x00,0x01,0x10,0x1D,0x2D,0x10,0x11,0xFF,0x6B,0x8B,0x84,0x02,0x8E,0x97,0x9D,0x81,0x9A,0xEE,0xF7,0x85,0x6F,0x12,0x3D,0xCB,0xFC,0x81,0xC3,0xF4,0x07,0x02,0xA9,0xAE,0xB0,0xBF,0xA9,0xF7,0xF4,0x24,0xFE,0xFF,0xB1,0x07,0x05,0xF0,0x31,0x3A,0x1E,0xA1,0x1E,0x0F,0x13,0x4E,0x02,0x1D,0x3D,0x2C,0x00,0xF1,0x1F,0x1F,0x1F,0x3F,0x1F,0x0E,0x3F,0x0E,0x2F,0x21,0x8D,0x81,0x00,0x00,0x10,0xF3,0xFE,0x1F,0xF0,0xF1,0xBF,0x85,0xFF,0x21,0xE2,0xE2,0xE0,0xE2,0xE2,0x0E,0x2F,0x13,0xC3,0x0E,0x2F,0xD2,0xF1,0x10,0x02,0xD1,0x0E,0x1F,0x02,0xE0,0x01,0x03,0xEE,0x2F,0xE4,0xC4,0xD2,0xFF,0x11,0xE3,0xC2,0xBB,0x90,0x01,0xE0,0xF2,0x1E,0x4F,0xFF,0x01,0xE0,0x1F,0x1E,0x12,0x0E,0x1F,0x2C,0x4E,0x1F,0x02,0xE3,0xE1,0xFF,0x01,0x1C,0x4E,0x02,0x0F,0x1F,0x0F,0x11,0xE1,0xFF,0x3D,0x95,0x88,0x01,0x2E,0xD2,0xF0,0x1F,0x10,0x00,0x1F,0x01,0xD3,0xF0,0x0E,0x01,0x87,0x86);
@target= decode_data (3, @source);
print "Source: (" . scalar @source . ") ";
for ($i= 0; $i < (scalar @source); $i++) {
print $source [$i] . " ";
}
print "\n\n";
print "Target: (" . scalar @target . ") ";
for ($i= 0; $i < (scalar @target); $i++) {
print "(" . $i . ")" . $target [$i] . " ";
}
print ".\n\n";
@target= (0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x85,0x81,0x83,0x82,0x84,0x84,0x84,0x84,0x83,0x85,0x84,0x84,0x83,0x81,0x80,0x80,0x82,0x84,0x85,0x85,0x86,0x85,0x85,0x85,0x84,0x86,0x84,0x84,0x81,0x82,0x82,0x84,0x85,0x86,0x85,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x86,0x8C,0x8A,0x8A,0x89,0x89,0x88,0x88,0x87,0x86,0x89,0x88,0x8A,0x8A,0x89,0x89,0x8A,0x8A,0x88,0x88,0x89,0x86,0x86,0x88,0x88,0x88,0x8A,0x8C,0x8D,0x8C,0x8C,0x8B,0x8B,0x89,0x89,0x88,0x87,0x87,0x88,0x8B,0x8B,0x8C,0x8A,0x8C,0x8C,0x8B,0x8B,0x8A,0x89,0x88,0x8A,0x8B,0x8C,0x8D,0x8D,0x8C,0x8C,0x8C,0x8B,0x8B,0x8A,0x89,0x88,0x87,0x87,0x88,0x89,0x8B,0x8A,0x8B,0x89,0x88,0x87,0x87,0x88,0x88,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x87,0x8E,0x97,0x9D,0x9A,0x88,0x75,0x69,0x6F,0x81,0x96,0xA8,0xB6,0xBF,0xC7,0xCB,0xC3,0xB7,0xAB,0xA6,0xA9,0xAE,0xB0,0xA9,0xA0,0x96,0x90,0x8C,0x8C,0x8B,0x88,0x84,0x7F,0x75,0x6C,0x63,0x61,0x5F,0x62,0x64,0x66,0x6B,0x71,0x7A,0x7D,0x81,0x83,0x7F,0x7C,0x7A,0x76,0x72,0x6D,0x69,0x68,0x6B,0x6C,0x6D,0x70,0x74,0x75,0x79,0x7A,0x7D,0x7C,0x7B,0x7A,0x78,0x77,0x77,0x76,0x76,0x75,0x75,0x74,0x76,0x77,0x79,0x7A,0x7B,0x7A,0x7C,0x7D,0x7E,0x7D,0x7E,0x7E,0x80,0x83,0x81,0x81,0x81,0x81,0x82,0x83,0x83,0x86,0x88,0x88,0x89,0x89,0x88,0x87,0x85,0x84,0x85,0x84,0x85,0x87,0x87,0x89,0x89,0x8B,0x8B,0x8B,0x89,0x89,0x87,0x87,0x87,0x85,0x85,0x84,0x84,0x87,0x86,0x88,0x8A,0x8A,0x8C,0x8D,0x8B,0x8B,0x8A,0x8A,0x8B,0x8C,0x8D,0x90,0x90,0x91,0x92,0x91,0x91,0x90,0x8F,0x90,0x8F,0x8E,0x8D,0x8D,0x8D,0x90,0x91,0x90,0x91,0x91,0x8F,0x91,0x8F,0x91,0x90,0x91,0x91,0x90,0x90,0x91,0x90,0x92,0x90,0x90,0x90,0x91,0x90,0x8F,0x8D,0x8D,0x8E,0x8D,0x90,0x92,0x93,0x93,0x93,0x94,0x93,0x92,0x92,0x91,0x91,0x8F,0x8E,0x8F,0x90,0x8F,0x8F,0x8E,0x8F,0x8C,0x8D,0x8C,0x8C,0x8B,0x8A,0x8B,0x8A,0x8C,0x8C,0x8D,0x8D,0x8C,0x8B,0x8B,0x8C,0x89,0x8A,0x89,0x88,0x89,0x8A,0x8A,0x8B,0x8B,0x8B,0x8A,0x8A,0x8B,0x8A,0x8A,0x89,0x87,0x88,0x86,0x88,0x89,0x8C,0x8D,0x8B,0x8B,0x8A,0x89,0x89,0x88,0x88,0x88,0x88,0x88,0x89,0x89,0x89,0x8A,0x88,0x89,0x89,0x89,0x89,0x87,0x87,0x86);
print "Target: (" . scalar @target . ") ";
for ($i= 0; $i < (scalar @target); $i++) {
print "(" . $i . ")" . $target [$i] . " ";
}
print ".\n";