汽车网红技术之EAVB

1 Misc
export LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8
VI行超过80字符长度提示：set colorcolumn=81
AVB: soft real-time
TSN: hard real-time, lookahead mechanism

2 untagged gPTP
2.1 switch gPTP
1) gateway - switch - SoC ETH
网关没有连接时，switch会发送SYNC和FOLLOW_UP报文，而当网关连接时，switch就不再发送这两个报文，而是由网关发送；PDELAY_REQ、PDELAY_RESP和PDELAY_RESP_FOLLOW_UP三个报文只会发生在gateway - switch和switch - SoC ETH之间。

switch的两种类型。
BC: switch收到gateway SYNC和FOLLOW_UP报文后，终止于switch并不再转发，switch重新生成SYNC和FOLLOW_UP报文发送给SoC ETH。
TC: switch收到gateway SYNC和FOLLOW_UP报文后，转发SYNC和FOLLOW_UP报文给SoC ETH。switch仅加上内部驻留延时到FOLLOW_UP报文CF字段的叫E2E，不但加上驻留延时还加上gateway - switch之间PDELAY_REQ测量所得路径延时的叫P2P。PDELAY_REQ等报文是由gateway - switch和switch - SoC ETH互相发送的，而DELAY_REQ只能由SoC ETH发送给gateway。

2）交换机延时
交换机延时包括两部分，路径延时（Link Delay）和驻留时间（Residence Time）。
路径延时使用PDELAY_REQ和PDELAY_RESP测量。
驻留时间等于Egress出口时间与Ingress入口时间的差值。

3) switch firmware gPTP
0x00：SYNC，ingress timestamp，每秒1次，master与slave之间的通信报文，这种模式交换机采用TC（relay）时钟，如果交换机直接发送SYNC和FOLLOW_UP报文给slave，那么交换机就采用BC时钟。
0x08：FOLLOW_UP，发送频率同Sync，消息中6个字节表示秒，4个字节表示纳秒，大端格式；Follow_Up报文中包含TLV。
0x02：PDELAY_REQ，egress timestamp，P表示Peer，带有P前缀的是交换机port与master或者slave之间的报文，需要交换机固件处理，用于P2P。
0x03：PDELAY_RESP，ingress timestamp，P表示Peer，带有P前缀的是交换机port与master或者slave之间的报文，需要交换机固件处理，用于P2P。
0x0a：PDELAY_RESP_FOLLOW_UP，P表示Peer，带有P前缀的是交换机port与master或者slave之间的报文，需要交换机固件处理，用于P2P。

SYNC报文以主时钟为起点，经过网络扩散到每个节点，在此过程中每个节点只需关注自己和上级节点的传输延时和频率同步，Bridge负责将中间路径的传输延时和驻留时间逐级累加到Correction字段（补偿），从而实现整个网络的同步。
Note: Hardware timestamp comes from two MAC_System_Time_X registers of DWMAC_v5.10a.

4）2018 55nm 88Q5050
交换机的每个port都有PHY地址，并且port号等于PHY地址。
88Q5050支持透明时钟，并且集成cortex M7和一个MAC，交换机内部P0连接到该MAC上，CM7运行gPTP协议栈，这也是交换机有MAC地址的由来。AVB交换机配置静态AVB组播MAC地址到ARL CAM（Address Resolution Logic）时，只需要配置接收该组播流的port mask（egress），而发送组播流的port mask（ingress）不需要配置。

88Q5050 firmware supports AVnu AutoCDS v1.4 profile (Automotive Certification Development Subgroup).
Need configure gPTP mc MAC to ARL for switch P0, P0 is connected to Cortex M7, otherwise switch gPTP stack does not handle PDELAY_REQ.

5) 28nm RTL905XB
offset-4 1-byte Domain Number of gPTP header means multiple gPTP instances.

2.2 Linux dynamic posix clock
udevadm monitor &
udevadm info -a /sys/class/ptp/ptp0
udevadm test /sys/class/ptp/ptp0

PTP start time: 1/1/1970 00:00:00 TAI
Linux下支持gPTP的网卡驱动会调用ptp_clock_register()创建一个gPTP类/sys/class/ptp和字符设备/dev/ptpN。
打开/dev/ptpN后使用宏FD_TO_CLOCKID(fd)将fd转换成clock id，然后就可以使用标准的posix clock API，诸如clock_gettime()、clock_settime()和clock_adjtime()等。

#define CLOCKFD 3
#define FD_TO_CLOCKID(fd)   ((~(clockid_t) (fd) << 3) | CLOCKFD)
#define CLOCKID_TO_FD(clk)  ((unsigned int) ~((clk) >> 3))

struct timeval tv;
clockid_t clkid;
int fd;

fd = open("/dev/ptp0", O_RDWR);
clkid = FD_TO_CLOCKID(fd);
clock_gettime(clkid, &tv);

2.3 SIOCSHWTSTAMP ioctl
Enable PTP timestamp through SIOCSHWTSTAMP ioctl.

2.4 Linux gPTP
2.4.1 xgmac v3.10a untagged gPTP
non-forwardable mc: 01-80-C2-00-00-0E
forwardable mc: 01-1B-19-00-00-00
MAC_RxQ_Ctrl1.PTPQ is used to select which DMA channel receives the PTP packets.
MAC_Timestamp_Control.TSENMACADDR is used to Enable MAC Address for PTP Packet Filtering.

2.4.2 socket timestamp
The gPTP time synchronization provides time of day, not media clock.
SO_TIMESTAMP: software timestamp, usec resolution
SO_TIMESTAMPNS: software timestamp, NS means nsec resolution
SO_TIMESTAMPING: hardware timestamp

2.4.3 gPTP ingress timestamp
SYNC/FOLLOW_UP: use the same seqID
PDELAY_RESP/PDELAY_RESP_FOLLOW_UP: use the same seqID
Linux recvmsg() with flags = 0 to get ingress timestamp.

2.4.4 gPTP egress timestamp
PDELAY_REQ: use the independent seqID
Linux recvmsg() with flags = MSG_ERRQUEUE to get egress timestamp.

2.5 DWMAC_v5.10a PPS
gPTP clock source.
7nm SA8155: 96MHz
8nm EA9: 26MHz

Two MAC_System_Time_X registers are directly accumulated every clock cycle of gPTP clock.

PPS（hw timer）的输入计数频率来自于gPTP时钟，设置PPS的寄存器MAC_PPSn_Interval和MAC_PPSn_Width可以产生特定频率的输出。

PPS supports MCGR and PPS mode. MCGR works with PTP digital rollover mode (MAC_Timestamp_Control.TSCTRLSSR = 1), refer to MAC_Timestamp_Control.bit9. When MCGR mode is enabled, MAC_PPS_Target_Time_Seconds is as a nano second comparing register. Take MCGR registers as an example.
MAC_PPS_Control
MAC_PPS_Target_Time_Seconds: Target_Time is only used for comparing with MAC_Presn_Time_ns
MAC_Presn_Time_ns
MAC_Presn_Time_updt

PTP kernel clk ftrace
/sys/kernel/debug/tracing/events/clk

3 DWMAC_v5.10a
3.1 CBS
3.1.1 DWMAC_v5.10a
DMA - Queues - Tx FIFO - CBS and TAS - MAC
PHY传输速率是1000Mbps，MAC和PHY之间采用RGMII接口，工作频率125MHz，那么每个时钟节拍传输8个bit（DDR）。
假如AVB传输占用带宽是70%，则8个bit中有5.6个bit用来传输AVB（IdleSlope = 5.6 x 1024 bits/s），剩余的2.4个bit用来传输Best Effort（SendSlope = 2.4 x 1024 bits/s）。SendSlope一般要小于等于IdleSlope。
HiCredit等于MaxInterferenceSize，突发流量的大小，单位为bits，每个调度间隔最大累积的bits。现在假设AVB帧排队时，总线上有其它帧正在传输，则AVB帧按照IdleSlope速率开始累积Credit，由于汽车应用属于Class C，其调度间隔是1.3333ms，从而对于1000Mbps网络，每个调度间隔HiCredit的最大值为1000Mbps / (1second / 1.333ms)。
LoCredit等于-MaxFrameSize，寄存器配置时采用补码，单位为bits，每个调度间隔最大减少的bits。现在假设AVB帧排队时，总线上没有其它帧等待传输，并且Credit的值刚好大于0，则AVB帧立刻开始传输，由于汽车应用属于Class C，其调度间隔是1.3333ms，从而本次调度间隔内，LoCredit可以减少到的最低值等于- (1000Mbps / (1second / 1.333ms)) 。由于HiCredit和LoCredit的单位都是bits，而网卡传输数据时是按照byte计算的，所以如果HiCredit和LoCredit的值不是整数个byte，需要取近似值。
补码转换为原码：符号位bit31不变，数值位按位取反，末位bit0再加1，即补码的补码等于原码。

3.1.2 88Q5050
QPri N Rate[15:0]: Unit Kbps
QPri N Hi Limit[13:0]: Unit bytes
The N is from 0 to 7.

3.2 DMA
3.2.1 MAC and queue
SA8155 includes 1 dwmac.
Samsung EA9 includes 2 XGMACs, one of XGMACs is shared between Linux and Android.
AMD NIC am79C90, after 6-byte source MAC, followed by 2-byte Length not EtherType, Length = [46, 1500]. DWC Ethernet appends padding automatically according to mti_crc_pad_ctrl_i signal.

DCB: Data Center Bridging, VERY IMPORTANT
MTL: MAC Transaction Layer
MTL_RxQ_DMA_Map0.bit4: DA-based DMA channel selection
MAC_RxQ_Ctrl1.bit[6:4]: gPTP Rx queue route
MAC_RxQ_Ctrl2.PSRQ0[7:0]: VLAN PCP filter

stmmac uses ring mode by default, can change it through module parameter chain_mode.
debugfs_create_file_unsafe() for debugfs mmap.
Linux kernel phys_to_virt() for DMA descriptors point to RAM physical address.
Linux kernel vmap() for mapping continuous RAM physical address to virtual address.

3.2.2 DMA
TD_BASE：描述符array物理基地址，地址offset是0x1114
TD_LEN：最大描述符个数 - 1，不是字节数；参考datasheet中的说明
TD_TAIL：描述符尾地址，如果TD_TAIL指向的地址对应描述符N，那么有效描述符到N - 1，当DMA引擎读取到TD_TAIL指针指向的描述符N时，自动停止进入suspend模式。软件再次写入TD_TAIL时，DMA会自动退出suspend模式并开始处理
cur_tx和cur_rx：软件需要定义一个尾指针，其值始终等于TD_TAIL地址指向的描述符N，当有数据需要传输时，根据cur_tx或者cur_rx，得到下一个可以使用的描述符的虚拟地址，填充描述符，填充好后，升级cur_tx或者cur_rx，同时将cur_tx或者cur_rx指向的描述符对应物理地址写入TD_TAIL

特别注意的是，正常的一个以太网发送报文对应一个DMA描述符，但是有时也会将一个以太网发送报文用多个DMA描述符表示，其中同一个报文的第一个DMA描述符中的字段FD的值为1，同一个报文的最后一个DMA描述符中的字段LD值为1，中间的DMA描述符的字段FD和LD的值都为0。譬如发送TSO报文时，第一个DMA发送描述符表示以太网的报文头（包括MAC header、IP header和TCP header），第二个DMA发送描述符表示TSO报文的payload。使能DMA描述符的时间戳Write-Back后，collision count的值都大于16，但是不影响功能，禁止时间戳Write-Back后collision count的值固定为2。DMA描述符的writeback时间戳就是gPTP时间戳，Write-Back timestamp is composed of two 32 bit，the unit of DES1 is second，the unit of DES0 is nanosecond。

3.2.3 Failed to reset the dma
7nm SA8155: RGMII functional clock is disabled
8nm EA9: switch hardware broken

3.3 time slot
3.3.1 time slot
software latency causes to miss time slot.

Class C规范中，每个CRF报文后面会跟着一条AVTP报文，AVTP报文时间戳 = CRF报文时间戳 + other + max_transit_time。将2次CRF报文之间的时间差1.3333ms分成64等分，每个等分是20.83us（= 1/48KHz），则每一个等分对应Audio codec 48KHz播放速率的一个采样点，那么AVTP报文中的时间戳会接近64等分中的其中某一个等分对应的时间戳，并且AVTP报文中的时间戳与某一个等分时间戳之差应该小于20.83us的5%。

Class C规范AVTP报文48KHz频率偏移计算：2个相邻AVTP报文中的时间戳相减获得delta，delta的倒数 x 每个AVTP报文中的sample个数64，计算出的值与48KHz有差异，差异小于等于5Hz。

3.3.2 Time-Based Scheduling
The launch time is a 32-bit value, where most-significant 8-bits represents the time in seconds and the rest 24-bits represent the time in 256 ns, which means each bit represents 256ns. So after read the nano seconds from register, need (nano_secs & 0xff) >> 8, then write the value to Enhanced Descriptor.
EDSE: DMA_CH(#i)_Tx_Control.bit28, Enhanced Descriptor Enable

How does Linux wake up real-time thread.
hrtimer_interrupt()
try_to_wake_up()
Get ETF delta time.
cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0
Choose max latency from cyclictest output result (unit: us).

3.3.3 EST
EST means Enhancements to Scheduled Traffic, IEEE 802.1Qbv-2015, which is for TSN. EST only works with PTP digital rollover mode (MAC_Timestamp_Control.TSCTRLSSR = 1), refer to MAC_Timestamp_Control.bit9.

For testing purpose, you can write a gate value 0x00 to gcl, then all of the Tx queues will be stopped, so you will fail to ping. Check MTL_EST_Status.bit2 for confirmation.

Base Time Register: upper 32-bits holds time in seconds and lower 32 bits hold time in nano seconds. It comes from gPTP time.
MAC_System_Time_Nanoseconds: PTP digital rollover mode maximum 999,999,999, seconds = MAC_System_Time_Nanoseconds / 1,000,000,000, seconds need to be added to MAC_System_Time_Seconds.
MTL_EST_GCL_Control: [19:8] for GCL row index.
Gate Control List: MTL_EST_GCL_Data, 8 bits Gate Controls (8 Transmit queues) and 24 bits Time Interval (in nano seconds, each bit represents 1ns). Get the depth of Gate Control List (1024) and width of the Time Interval (24 bits) from MAC_HW_Feature3. Time Interval shall be greater than one best effort MTU plus 20-byte overhead time.
LLR: List Length Register.
PTOV: ((1000000000 / ptp_rate) x 6)
Refer to 8.14 of DWMAC_v5.10a.

GMAC_HW_FEATURE3[24:23]: GCLWID, Width of Time Interval
GMAC_HW_FEATURE3[22:20]: GCLDEP, Depth of GCL
GMAC_HW_FEATURE3[19]: ESTSEL
MAC_TIMESTAMP_CONTROL[9]: Timestamp Digital or Binary Rollover Control
MTL_EST_GCL_CONTROL[17:8]: Gate Control List Address
MTL_EST_GCL_CONTROL[2]: GCRR, 1 for registers, 0 for GCL list
MTL_EST_GCL_CONTROL[1]: R1W0, Read 1, Write 0
MTL_EST_GCL_CONTROL[0]: SRWO, Start Read/Write Op
MTL_EST_CONTROL[31:23]: PTOV, PTP Time Offset Value
MTL_EST_CONTROL[1]: SSWL, Switch to software owned list, Store table
MTL_EST_CONTROL[0]: EEST, Enable EST

3.3.4 Frame Preemption
抢占式MAC层分为Express MAC和Preemptable MAC，为了保护高优先级队列的优先传输，它使得以太网帧最小到进行64字节的传输，如果高优先级需要传输时，前面有一个低优先级队列正在传输数据，而且，它还特别长，占用较多的传输时间，它就可以抢占方式获得传输，而被抢占的则会拆分，然后在下一个时段继续传输。当然，对于被抢占帧而言，其传输必须以完整的帧方式汇合，这中间涉及到重新分包（mPacket）和CRC校验，然后再到MAC聚合层进行重新封装为完整包。

3.4 Ring Buffer
// PCAP: Audio Data, Sample Chunk N
// la0: 96 payload buf, MTU 1536-byte (= 1024 + 512)
// lv0: 512 payload buf, MTU 1536-byte (= 1024 + 512)
// tv0 hdr phy addr: phy_base + structure_member_offset
common structure
ta0 stream structure
ta0 hdr buf_0
...
ta0 hdr buf_95
ta0 payload buf_0
...
ta0 payload buf_95
...
common structure
tv0 stream structure
tv0 hdr buf_0
...
tv0 hdr buf_63
tv0 payload buf_0
...
tv0 payload buf_511

configuration arguments sequence.
arg1: stream_name
arg2: stream_id

3.5 XGMAC_3.20a
X means Latin letter 10.
PHY: USXGMII
Tx timestamp in XGMAC comes from MAC instead of descriptors.
MAC_Timestamp_Status.TXTSC
MAC_Tx_Timestamp_Status_PktID
MAC_Tx_Timestamp_Status_Nanoseconds
MAC_Tx_Timestamp_Status_Seconds

4 Macbook AVDECC
IEEE 1722.1 provides the Audio Video Discovery, Enumeration, Connection management, and Control (AVDECC) protocol for AVB devices. AVDECC is not used by Avnu Automotive Profile.
AEM: AVDECC Entity Model
ADP: AVDECC Discovery Protocol
ACMP: AVDECC Connection Management Protocol
AECP: AVDECC Enumeration and Control Protocol

avbutil is used for the management of AVB features and settings. It enables control of AVB entities using the AVDECC (IEEE 1722.1) protocol.

The avbutil command is included in macOS since macOS Sierra, which was the first macOS release supporting AVB. Coincidentally, it was also the first macOS release that supported Thunderbolt, which happens to be a corresponding requirement for supporting AVB on Mac hardware.

github avdecc
Macbook avbdeviced

5 Linux
5.1 iperf3 stress testing
head unit server:
ifconfig eth0 192.168.1.2 up
./iperf3 -i 10 -s
Linux PC client:
iperf3 -i 10 -t 60 -c 192.168.1.2

5.2 KBA
filp->f_path.dentry->d_iname
kernel中vma->vm_pgoff等于mmap()函数最后一个参数offset >> PAGE_SHIFT
DEFINE_IDA(sd_index_ida)
Linux read_poll_timeout_atomic()
dcache invalid API：检查对应内存cache line的valid bit。如果valid bit为1，置为0。这样后续如果CPU读这段内存，就不会使用dcache中的数据了。

5.3 debug
1) high CPU load, API kstat_irqs_cpu(irq, cpu).
2) kernel virtual address prefix 0xffff_ff8 1-bit flip, LPDDR5 has 8bit ECC for each write and read.
3) recvmsg() hang. This issue was reproduced with basic switch firmware, that means switch firmware without gPTP support.
4) signal=4
SIGILL = 4
kill -s 4 <pid>

# include <execinfo.h>
static void sigill_handler(int signo)
{
    const int len = 256;
    void *buffer[len];
    int nptrs;
    char **strings;
    int i;

    nptrs = backtrace(buffer, len);
    strings = backtrace_symbols(buffer, nptrs);
    printf("received %d, nptrs: %d\n",
        signo, nptrs);
    if (strings) {
        for (i = 0; i < nptrs; i++)
            printf("[%02d] %s\n",
                i, strings[i]);
        free(strings);
    }
    // signal(signo, SIG_DFL);
    // raise(signo);
}

backtrace_symbols() dumps the call trace, [02] ./my_main(+0xbcc) [0x5572bc0bcc], 0xbcc is the address offset, search it directly from objdump file.
$OBJDUMP -l -C -S my_main > my_main.asm

Default core dump, must use sudo bash -c 'shell command' to run coredump configuration.
sudo bash -c 'echo "/data/core.%e.%p" > \
/proc/sys/kernel/core_pattern'
ulimit -c <size>: unit is block (= 512bytes)
ulimit -c unlimited
gdb main core.main.19188

5) signal=6, caused by Unable to handle kernel paging request at virtual address.
6) signal=11
Cannot access memory at address 0xNNN.
Debug stack overflow
-fsanitize=address -fno-omit-frame-pointer
因为gPTP使用的是L2报文，因此使用recvmsg()和sendmsg()接收和发送gPTP报文时，需要将struct msghdr.msg_name和struct msghdr.msg_namelen分别设置为NULL和0，不然Kernel可能不检查msg_namelen的值，而直接向msg_name写入超过16个字节的数据，导致用户空间stack overflow。
7) Debug kernel stack overflow
-fstack-protector
CONFIG_CC_STACKPROTECTOR=y
POSIX timer_create based on futex API.
8) smmu cb=32 crash. Rx Descriptor OWN bit is 1 but RDES0 is 0. stmmac_rx_refill() for Rx.
9) transmit queue 0 timed out, re-initialize RGMII before resume exits.
10) unclocked access.
11) dcache invalidate and writeback.
12) CRC32 Hash Multicast: MAC_Hash_Table_RegN, get hash table size from MAC_HW_Feature1. MSB 3-bit + LSB 5-bit, 3-bit index specific register, 5-bit index the bit within the register, refer to ~ether_crc() in xgmac.c.
13) Rx trailing bytes 04 28 00 00 40 00 00 00, and 40 00 00 00 is the 4-byte CRC, DMA Rx buffer memory alignment.
14) Wireshark
Use "ieee1722.subtype == 0x03" to filter AVB video.
Wireshark: File - Export Specified Packets...
Windows CMD: copy /b 1.pcap+2.pcap 3.pcap
15) DWMAC RGMII DLL (Delay Loop Lock) lock timeout because of switch RGMII issue.
在RGMII MAC-MAC连接的情况下，一个MAC的TX将连接到另一个MAC的RX，这种连接称为Reverse RGMII。在RGMII MAC-PHY连接的情况下，MAC的TX和RX信号连接到PHY的各个TX和RX信号。
16) Video Tx 0-length buffer.
17) DWMAC and switch RGMII RXC Issue
Use Linux kernel call_usermodehelper("/bin/ls", argv, envp, UMH_WAIT_PROC) to run user space binary.

5.4 Suspend-To-RAM
drivers/base/power/main.c

6 Reverse MII
Reverse MII by AMD in 1997. This mode is used for ethernet Media Converter.
The MII signals are cross connected: RX output signals from one PHY are connected to the TX inputs on the other PHY. The 100Base-T1 PHY TXC and RXC clock signals become inputs in Reverse MII mode.
88EA1512
88Q2110

7 Abbreviations
100Base-T1 MediaConverter：车载以太网和电脑之间物理层转换
802.1as：AVB timing Synchronization
802.1Q：Q表示Queue
802.1Qav：Qav表示Queue AVB
AAF: AVTP Audio Format
AM824 Protocol: FireWire "Adaptation Layer" for MIDI
AP：AUTOSAR Adaptive Platform
CLOCK_TAI：International Atomic Time，原子钟
CP：AUTOSAR Classic Platform
CRF: AVTP Clock Reference Format
DWC sbd_intr: sideband interrupt
EGR_TSFIFO_0：VSC8490 Egress Timestamp FIFO
enp0s10: ethernet (en), prefix 0 (p0), slot 10 (s10). PCI BDF and physical/geographical location of the connector. systemd udev-builtin-net_id.c, /etc/systemd/network/99-default.link
HPET：High Precision Event Timer
IVN: In Vehicle Network
Linux usr: Unix System Resource
MACsec：Media Access Control Security
Milan: Avnu Media-integrated local area network, for AVB interoperability
MRQC：IGB Multiple Receive Queues Control
PHC：PTP Hardware Clock
PIT：Programmable Interval Timer
ppb：parts per billion
ppm：parts per million
PPS：Pulse Per Second
SoP：Start of Packet
TC9560: Toshiba Companion-chip for Ethernet AVB
ToD：Time-of-Day，gettimeofday()
TSC：Time Stamp Counter
TSN Box: TSN Systems
VSC7428-02: Vitesse Semiconductor Corporation
XMOS: ex-INMOS. represented both the International founding of the company as well as the basis for VLSI components: n-channel MOS, or NMOS technology. Thus, it became known as International MOS or INMOS