CLOCK_MONOTONIC datagram timestamps by the kernel

CLOCK_MONOTONIC datagram timestamps by the kernel

[John]

Hello,

It is possible to ask Linux to timestamp incoming datagrams when they
are received, then to retrieve this timestamp with an ioctl command or
a recvmsg call (which would save one round trip to kernel space).

SIOCGSTAMP
     Return a struct timeval with the receive timestamp of the last
packet passed to the user. This is useful for accurate round trip time
measurements. See setitimer(2) for a description of struct timeval.

As far as I understand, this timestamp is given by the CLOCK_REALTIME
clock. I would like to get the timestamp given by a different clock: the
CLOCK_MONOTONIC clock.

In other words, I would like the kernel to do the equivalent of

   struct timespec spec;
   clock_gettime(CLOCK_MONOTONIC, &spec)

for each datagram the system receives, as soon as it is received.

Is there a way to achieve that?

Is there a different ioctl perhaps? (I don't think so.)

Regards.

Re: CLOCK_MONOTONIC datagram timestamps by the kernel

[John]

John wrote:

> It is possible to ask Linux to timestamp incoming datagrams when they
> are received, then to retrieve this timestamp with an ioctl command or
> a recvmsg call (which would save one round trip to kernel space).
> 
> SIOCGSTAMP
>     Return a struct timeval with the receive timestamp of the last
> packet passed to the user. This is useful for accurate round trip time
> measurements. See setitimer(2) for a description of struct timeval.
> 
> As far as I understand, this timestamp is given by the CLOCK_REALTIME
> clock. I would like to get the timestamp given by a different clock:
> the CLOCK_MONOTONIC clock.
> 
> In other words, I would like the kernel to do the equivalent of
> 
>   struct timespec spec;
>   clock_gettime(CLOCK_MONOTONIC, &spec)
> 
> for each datagram the system receives, as soon as it is received.
> 
> Is there a way to achieve that?
> 
> Is there a different ioctl perhaps? (I don't think so.)

I've been reading the following documents:

Clocks and Timers
http://www.opengroup.org/onlinepubs/009695399/xrat/xsh_chap02.html#tag_03_02_08_18

Rationale for the Monotonic Clock
http://www.opengroup.org/onlinepubs/009695399/xrat/xsh_chap02.html#tag_03_02_08_19

"For those applications that use time services to achieve realtime 
behavior, changing the value of the clock on which these services rely 
may cause erroneous timing behavior. For these applications, it is 
necessary to have a monotonic clock which cannot run backwards, and 
which has a maximum clock jump that is required to be documented by the 
implementation. Additionally, it is desirable (but not required by IEEE 
Std 1003.1-2001) that the monotonic clock increases its value uniformly. 
This clock should not be affected by changes to the system time; for 
example, to synchronize the clock with an external source or to account 
for leap seconds. Such changes would cause errors in the measurement of 
time intervals for those time services that use the absolute value of 
the clock."

clock and timer functions
http://www.opengroup.org/onlinepubs/009695399/functions/clock_settime.html

"If the value of the CLOCK_REALTIME clock is set via clock_settime(), 
the new value of the clock shall be used to determine the time of 
expiration for absolute time services based upon the CLOCK_REALTIME 
clock. This applies to the time at which armed absolute timers expire. 
If the absolute time requested at the invocation of such a time service 
is before the new value of the clock, the time service shall expire 
immediately as if the clock had reached the requested time normally."

"The effect of setting a clock via clock_settime() on armed per-process 
timers associated with a clock other than CLOCK_REALTIME is 
implementation-defined."


Here's some background on what I'm trying to do:

My app expects a stream of UDP datagrams containing compressed video.
These datagrams have been time stamped by the source using a high 
resolution clock.

The video stream is played out in real-time. I buffer several packets, 
then repeatedly arm a one-shot timer (with TIMER_ABSTIME set) to wait 
until it's time to send the oldest packet.

AFAIU, if I use the CLOCK_REALTIME clock in my app, and if the sysadmin 
changes the date, hell will break loose in my app. Hence my need for 
CLOCK_MONOTONIC.


On a related note, I see that do_gettimeofday() in kernel/timer.c 
actually gets a struct timespec from __get_realtime_clock_ts() then 
throws bits away in the ns to us conversion. The integer division is 
also a waste of cycles in my case, as I want the result in ns.

I suppose that if I change sock_get_timestamp() in core/sock.c to call 
__get_realtime_clock_ts() instead of do_gettimeofday() I'll break 50 
billion applications (ping? traceroute?) that expect a struct timeval?

Anyway, thanks for reading this far.

Regards.

Re: CLOCK_MONOTONIC datagram timestamps by the kernel

[Andi Kleen]

Mr Anonymous,

> 
> My app expects a stream of UDP datagrams containing compressed video.
> These datagrams have been time stamped by the source using a high 
> resolution clock.

Why do you need another time stamp them? 
> 
> The video stream is played out in real-time. I buffer several packets, 
> then repeatedly arm a one-shot timer (with TIMER_ABSTIME set) to wait 
> until it's time to send the oldest packet.
> 
> AFAIU, if I use the CLOCK_REALTIME clock in my app, and if the sysadmin 
> changes the date, hell will break loose in my app.

Only if your app cannot handle time going backwards.

> I suppose that if I change sock_get_timestamp() in core/sock.c to call 
> __get_realtime_clock_ts() instead of do_gettimeofday() I'll break 50 
> billion applications (ping? traceroute?) that expect a struct timeval?

Not that many, but some probably. 

Letting the kernel do the time stamping is usually quite useless anyways. 
You can as well do it in your application when you receive it. After all
you're interested in when you can read the packet in your app, 
not when the driver processes it.

The kernel time stamping is mainly for sniffing applications that want
to know exactly what's going on on the wire. But even then it's fairly
arbitary because it can be a long time between the packet arriving 
on the PHY and being processed by the driver.

-Andi

Re: CLOCK_MONOTONIC datagram timestamps by the kernel

[John]

Andi Kleen wrote:

> John wrote:
> 
>> My app expects a stream of UDP datagrams containing compressed video.
>> These datagrams have been time stamped by the source using a high
>> resolution clock.
> 
> Why do you need another time stamp then?

Source and receiver do not share a common clock. Thus, I have to monitor 
clock skew. If I simply use the time stamps produced by the source, my 
buffer will eventually underflow or overflow.

As far as I have seen, clock skew is on the order of ±10^-5, i.e. when 
the source thinks 100000 seconds have elapsed, the receiver thinks 
100001 seconds have elapsed.

>> The video stream is played out in real-time. I buffer several packets,
>> then repeatedly arm a one-shot timer (with TIMER_ABSTIME set) to wait
>> until it's time to send the oldest packet.
>>
>> AFAIU, if I use the CLOCK_REALTIME clock in my app, and if the sysadmin 
>> changes the date, hell will break loose in my app.
> 
> Only if your app cannot handle time going backwards.

My app cannot handle time going backwards.

The algorithm looks like:

/* Start transmission, record start date */
NEXT_DEADLINE = now;

while ( 1 )
{
   send(oldest_packet);
   recv_available_packets();
   /* compute next deadline */
   NEXT_DEADLINE += time_to_wait_until_next_packet();
   sleep_until(NEXT_DEADLINE);
}

If the clock changes under me, I'm toast.

>> I suppose that if I change sock_get_timestamp() in core/sock.c to call 
>> __get_realtime_clock_ts() instead of do_gettimeofday() I'll break 50 
>> billion applications (ping? traceroute?) that expect a struct timeval?
> 
> Not that many, but some probably.

Another option would be to change sock_get_timestamp() to call 
ktime_get_ts() instead of do_gettimeofday() and convert ns to us.

i.e. I still return a struct timeval, but using CLOCK_MONOTONIC.

> Letting the kernel do the time stamping is usually quite useless anyways. 
> You can as well do it in your application when you receive it. After all
> you're interested in when you can read the packet in your app, 
> not when the driver processes it.

I use clock_nanosleep() to sleep until it's time to send the next 
packet. I check for packets when I wake up. I need the kernel to time 
stamp the packets because I was sleeping when they reached the system.

I try to do it this way because I read this:

http://rt.wiki.kernel.org/index.php/High_resolution_timers

"Note that (clock_)nanosleep functions do not suffer from this problem 
as the wakeup function at timer expiry is executed in the context of the 
high resolution timer interrupt. If an application is not using 
asynchronous signal handlers, it is recommended to use the 
clock_nanosleep() function with the TIMER_ABSTIME flag set instead of 
waiting for the periodic timer signal delivery. The application has to 
maintain the absolute expiry time for the next interval itself, but this 
is a lightweight operation of adding and normalizing two struct timespec 
values. The benefit is significantly lower maximum latencies (~50us) and 
less OS overhead in general."

Regards.

Re: CLOCK_MONOTONIC datagram timestamps by the kernel

[John]

John wrote:

> Andi Kleen wrote:
> 
>> John wrote:
>>
>>> My app expects a stream of UDP datagrams containing compressed video.
>>> These datagrams have been time stamped by the source using a high
>>> resolution clock.
>>
>> Why do you need another time stamp then?
> 
> Source and receiver do not share a common clock. Thus, I have to monitor
> clock skew. If I simply use the time stamps produced by the source, my
> buffer will eventually underflow or overflow.
> 
> As far as I have seen, clock skew is on the order of ±10^-5, i.e. when
> the source thinks 100000 seconds have elapsed, the receiver thinks
> 100001 seconds have elapsed.

I've considered running NTP to synchronize the clocks, but I'm afraid it 
wouldn't help, for several reasons:

1) The 40.5 MHz clock used to timestamp datagrams at the source is 
located on a PCI board. I have read access to a 32-bit counter that is 
incremented every cycle (somewhat like the TSC on x86).

2) Source and receiver are inside a VPN, and do not have access to the 
Internet. I don't think I can keep NTP happy if it can't talk to any NTP 
servers.

(I'm not snipping the rest of the message in case someone is willing
to comment.)

>>> The video stream is played out in real-time. I buffer several packets,
>>> then repeatedly arm a one-shot timer (with TIMER_ABSTIME set) to wait
>>> until it's time to send the oldest packet.
>>>
>>> AFAIU, if I use the CLOCK_REALTIME clock in my app, and if the 
>>> sysadmin changes the date, hell will break loose in my app.
>>
>> Only if your app cannot handle time going backwards.
> 
> My app cannot handle time going backwards.
> 
> The algorithm looks like:
> 
> /* Start transmission, record start date */
> NEXT_DEADLINE = now;
> 
> while ( 1 )
> {
>   send(oldest_packet);
>   recv_available_packets();
>   /* compute next deadline */
>   NEXT_DEADLINE += time_to_wait_until_next_packet();
>   sleep_until(NEXT_DEADLINE);
> }
> 
> If the clock changes under me, I'm toast.
> 
>>> I suppose that if I change sock_get_timestamp() in core/sock.c to
>>> call __get_realtime_clock_ts() instead of do_gettimeofday() I'll
>>> break 50 billion applications (ping? traceroute?) that expect a
>>> struct timeval?
>>
>> Not that many, but some probably.
> 
> Another option would be to change sock_get_timestamp() to call
> ktime_get_ts() instead of do_gettimeofday() and convert ns to us.
> 
> i.e. I still return a struct timeval, but using CLOCK_MONOTONIC.
> 
>> Letting the kernel do the time stamping is usually quite useless
>> anyways. You can as well do it in your application when you receive
>> it. After all you're interested in when you can read the packet in
>> your app, not when the driver processes it.
> 
> I use clock_nanosleep() to sleep until it's time to send the next
> packet. I check for packets when I wake up. I need the kernel to time
> stamp the packets because I was sleeping when they reached the system.
> 
> I try to do it this way because I read this:
> 
> http://rt.wiki.kernel.org/index.php/High_resolution_timers
> 
> "Note that (clock_)nanosleep functions do not suffer from this problem
> as the wakeup function at timer expiry is executed in the context of the
> high resolution timer interrupt. If an application is not using
> asynchronous signal handlers, it is recommended to use the
> clock_nanosleep() function with the TIMER_ABSTIME flag set instead of
> waiting for the periodic timer signal delivery. The application has to
> maintain the absolute expiry time for the next interval itself, but this
> is a lightweight operation of adding and normalizing two struct timespec
> values. The benefit is significantly lower maximum latencies (~50us) and
> less OS overhead in general."