Upgrade to nuttx 9.1.0+ by davids5 · Pull Request #15139 · PX4/PX4-Autopilot

davids5 · 2020-06-18T17:11:06Z

@dagar FYI: @MaEtUgR @TSC21

I am adding the todo list we discussed yesterday. All Please add what I am missing.

Investigate SD error messages on debug config on FMUv5
PR is in https://github.com/apache/incubator-nuttx/pull/1741/files
vet FMU1062
run serial_test on DMA and non DMA UxARTS
Run the I2C killer on F4, F7, H7
- F4
- F7
- H7 Passes with Critical STM32H7 Interrupt Storm on I2C NuttX#117
  ~~H7 Is failing~~

Add the BDMA (def config, like map must have ".sram4") to the H7 FMUs and test SPI6
Add pass/fail criteria on sd_bench on test rack and document card ageing.
resolve Windows build support issues
Test, Test Test,
Did I mention Test? :)

after this comes in

Rework ROMFS to use source (.) and fix the paths to be relative on nuttx/linux see c99ff32 and back out excludes from shell_check
Update and Merge these 1afb7d9 and 60a97d8

davids5 · 2020-06-19T23:00:54Z

@dagar - let's see how the test rack feels about this, that we should start testing!

dagar · 2020-06-19T23:16:11Z

@dagar - let's see how the test rack feels about this, that we should start testing!

I think they're all in the same hardfault loop. http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-hardware/detail/pr-nuttx-9.1.0-/5/pipeline/517

davids5 · 2020-06-23T13:32:41Z

@dagar - prior to this passed on the HW rack without SD card working. Can we add 'sd_bench' ?

My Plan is to rippled the last 29 commits to the board configs once tested on each device.

davids5 · 2020-08-12T21:16:28Z

@dagar - I was able to build the nuttx CI list locally. The mac flock should get us the rest of the way.

davids5 · 2020-08-12T21:49:58Z

@dagar We are going to need GCC 9.3.1 also

davids5 · 2020-08-13T12:21:43Z

Issues list:

Need gcc 9.3.1 to fix V2
Version is incorrect

FW git-branch: pr-nuttx-9.1.0-
OS: NuttX
OS version: Release 8.2.0 (134349055)
```'

davids5 · 2020-08-13T13:04:10Z

V3: 2.4.6 mro SYS_AUTOSTART 4001

uorb top

KTOPIC NAME                     INST #SUB RATE #Q SIZE
�[Kactuator_armed                    0    6    2  1   24 
�[Kactuator_controls_0               0    6  400  1   48 
�[Kactuator_controls_3               0    2   45  1   48 
�[Kactuator_outputs                  0    3   45  1   80 
�[Kactuator_outputs                  1    2   79  1   80 
�[Kadc_report                        0    1  100  1   96 
�[Kbattery_status                    0    6  100  1  112 
�[Kcommander_state                   0    1    2  1   16 
�[Kcpuload                           0    4    1  1   16 
�[Kestimator_innovation_test_ratios  0    1  100  1  144 
�[Kestimator_innovation_variances    0    1  100  1  144 
�[Kestimator_innovations             0    1  100  1  144 
�[Kestimator_sensor_bias             0    4  100  1   56 
�[Kestimator_status                  0    4  100  1  288 
�[Kinput_rc                          0    4   45  1   72 
�[Kmanual_control_setpoint           0    5   45  1   72 
�[Kmultirotor_motor_limits           0    2   45  1   16 
�[Kpx4io_status                      0    1    1  1  128 
�[Krate_ctrl_status                  0    1  400  1   24 
�[Ksafety                            0    2    1  1   16 
�[Ksensor_accel                      0    4  400  8   48 
�[Ksensor_accel                      1    3  755  8   48 
�[Ksensor_baro                       0    2   75  1   32 
�[Ksensor_combined                   0    3  200  1   48 
�[Ksensor_gyro                       0    4  400  8   40 
�[Ksensor_gyro                       1    3  741  8   40 
�[Ksensor_mag                        0    3   96  1   48 
�[Ksensor_mag                        1    3  124  1   48 
�[Ksensor_preflight                  0    1  200  1   24 
�[Ksystem_power                      0    2  100  1   24 
�[Ktelemetry_status                  0    2    1  1  104 
�[Ktelemetry_status                  1    2    1  1  104 
�[Kvehicle_acceleration              0    1  200  1   32 
�[Kvehicle_air_data                  0    8   18  1   40 
�[Kvehicle_angular_acceleration      0    1  400  1   32 
�[Kvehicle_angular_velocity          0    5  400  1   32 
�[Kvehicle_attitude                  0    6  200  1   48 
�[Kvehicle_attitude_setpoint         0    4  200  1   64 
�[Kvehicle_control_mode              0    9    2  1   32 
�[Kvehicle_gps_position              0    5    5  1  104 
�[Kvehicle_imu                       0    3  200  1   56 
�[Kvehicle_imu                       1    3  192  1   56 
�[Kvehicle_imu_status                0    4   10  1   56 
�[Kvehicle_imu_status                1    4   10  1   56 
�[Kvehicle_land_detected             0    9    1  1   24 
�[Kvehicle_local_position            0   11  100  1  160 
�[Kvehicle_magnetometer              0    3  124  1   24 
�[Kvehicle_odometry                  0    1  100  1  256 
�[Kvehicle_rates_setpoint            0    4  200  1   32 
�[Kvehicle_status                    0   17    2  1   56 
�[Kvehicle_status_flags              0    1    2  1   40 
�[0J�[H�[Kupdate: 1s, num topics: 68
�[KTOPIC NAME                     INST #SUB RATE #Q SIZE
�[Kactuator_armed                    0    6    2  1   24 
�[Kactuator_controls_0               0    6  399  1   48 
�[Kactuator_controls_3               0    2   44  1   48 
�[Kactuator_outputs                  0    3   44  1   80 
�[Kactuator_outputs                  1    2   81  1   80 
�[Kadc_report                        0    1  100  1   96 
�[Kbattery_status                    0    6  100  1  112 
�[Kcommander_state                   0    1    2  1   16 
�[Kcpuload                           0    4    2  1   16 
�[Kestimator_innovation_test_ratios  0    1  100  1  144 
�[Kestimator_innovation_variances    0    1  100  1  144 
�[Kestimator_innovations             0    1  100  1  144 
�[Kestimator_sensor_bias             0    4  100  1   56 
�[Kestimator_status                  0    4  100  1  288 
�[Kinput_rc                          0    4   44  1   72 
�[Kmanual_control_setpoint           0    5   44  1   72 
�[Kmultirotor_motor_limits           0    2   44  1   16 
�[Kpx4io_status                      0    1    1  1  128 
�[Krate_ctrl_status                  0    1  399  1   24 
�[Ksafety                            0    2    1  1   16 
�[Ksensor_accel                      0    4  399  8   48 
�[Ksensor_accel                      1    3  753  8   48 
�[Ksensor_baro                       0    2   74  1   32 
�[Ksensor_combined                   0    3  200  1   48 
�[Ksensor_gyro                       0    4  399  8   40 
�[Ksensor_gyro                       1    3  736  8   40 
�[Ksensor_mag                        0    3   95  1   48 
�[Ksensor_mag                        1    3  124  1   48 
�[Ksensor_preflight                  0    1  200  1   24 
�[Ksystem_power                      0    2  100  1   24 
�[Ktelemetry_status                  0    2    1  1  104 
�[Ktelemetry_status                  1    2    1  1  104 
�[Kvehicle_acceleration              0    1  200  1   32 
�[Kvehicle_air_data                  0    8   18  1   40 
�[Kvehicle_angular_acceleration      0    1  399  1   32 
�[Kvehicle_angular_velocity          0    5  399  1   32 
�[Kvehicle_attitude                  0    6  200  1   48 
�[Kvehicle_attitude_setpoint         0    4  200  1   64 
�[Kvehicle_control_mode              0    9    2  1   32 
�[Kvehicle_gps_position              0    5    5  1  104 
�[Kvehicle_imu                       0    3  200  1   56 
�[Kvehicle_imu                       1    3  192  1   56 
�[Kvehicle_imu_status                0    4   10  1   56 
�[Kvehicle_imu_status                1    4   10  1   56 
�[Kvehicle_land_detected             0    9    1  1   24 
�[Kvehicle_local_position            0   11  100  1  160 
�[Kvehicle_magnetometer              0    3  124  1   24 
�[Kvehicle_odometry                  0    1  100  1  256 
�[Kvehicle_rates_setpoint            0    4  200  1   32 
�[Kvehicle_status                    0   17    2  1   56 
�[Kvehicle_status_flags              0    1    2  1   40

top

�[KProcesses: 26 total, 6 running, 20 sleeping, max FDs: 15
�[KCPU usage: 66.93% tasks, 5.26% sched, 27.81% idle
�[KDMA Memory: 5120 total, 1024 used 1024 peak
�[KUptime: 505.655s total, 169.868s idle
�[H�[K PID COMMAND                   CPU(ms) CPU(%)  USED/STACK PRIO(BASE) STATE FD
�[K   0 Idle Task                  170258 28.077   244/  512   0 (  0)  READY  3
�[K   1 hpwork                          0  0.000   392/ 1284 249 (249)  w:sig  3
�[K   2 lpwork                          0  0.000   360/ 1636  50 ( 50)  w:sig  3
�[K   3 init                          881  0.000  2264/ 2948 100 (100)  w:sem  3
�[K   4 wq:manager                      1  0.000   416/ 1276 255 (255)  w:sem  3
�[K 447 mavlink_rcv_if0              2178  0.503  2520/ 4092 175 (175)  w:sem  4
�[K  18 dataman                       165  0.000   784/ 1220  90 ( 90)  w:sem  4
�[K  20 wq:lp_default                 416  0.143   944/ 1716 205 (205)  w:sem  3
�[K  28 wq:hp_default                8382  2.159  1224/ 1916 240 (240)  w:sem  3
�[K 143 wq:I2C2                      1084  0.215   752/ 1492 245 (245)  w:sem  3
�[K 151 wq:SPI1                     91563 18.286  1496/ 2356 253 (253)  w:sem  3
�[K 190 wq:I2C1                      5907  1.223   976/ 1492 246 (246)  w:sem  3
�[K 203 wq:nav_and_controllers      95035 19.078  5296/ 7220 241 (241)  w:sem  3
�[K 204 wq:rate_ctrl                15938  3.167  1192/ 1684 255 (255)  w:sem  3
�[K 211 commander                   11117  1.871  1304/ 3236 140 (140)  w:sig  6
�[K 213 commander_low_prio             13  0.000   816/ 3020  50 ( 50)  w:sem  6
�[K 221 mavlink_if0                 33902  6.407  1856/ 2596 100 (100)  READY  4
�[K 293 gps                          2038  0.431   984/ 1700 205 (205)  w:sem  4
�[K 337 mavlink_if1                 10761  2.159  1904/ 2508 100 (100)  w:sig  4
�[K 342 mavlink_rcv_if1              2505  0.503  2544/ 4092 175 (175)  w:sem  4
�[K 368 px4io                       19598  4.319  1120/ 1500 237 (237)  w:sem  4
�[K 685 log_writer_file                 0  0.000   368/ 1188  60 ( 60)  w:sem  3
�[K 614 wq:attitude_ctrl             6257  1.223  1360/ 1660 242 (242)  w:sem  3
�[K 622 navigator                     676  0.143  1112/ 1788 105 (105)  w:sem  5
�[K 681 logger                       2988  0.647  2160/ 3660 230 (230)  READY  3
�[K1127 top                           112  3.959  1248/ 2052 239 (239)  RUN    3

work_queue status

nsh> �[Kwork_queue status

Work Queue: 8 threads                        RATE        INTERVAL
|__ 1) wq:rate_ctrl    
|   |__ 1) mc_rate_control               399.3 Hz         2504 us
|   \__ 2) vehicle_angular_velocity      399.1 Hz         2506 us
|__ 2) wq:SPI1         
|   |__ 1) l3gd20                       1398.5 Hz          715 us (715 us)
|   |__ 2) lsm303d                       952.4 Hz         1050 us (1050 us)
|   |__ 3) mpu6000                       399.2 Hz         2505 us
|   \__ 4) ms5611                         99.2 Hz        10081 us
|__ 3) wq:I2C1         
|   \__ 1) hmc5883                       124.2 Hz         8049 us
|__ 4) wq:I2C2         
|   \__ 1) rgbled                         38.8 Hz        25799 us
|__ 5) wq:attitude_ctrl
|   \__ 1) mc_att_control                199.7 Hz         5008 us
|__ 6) wq:nav_and_controllers
|   |__ 1) ekf2                          199.7 Hz         5008 us
|   |__ 2) land_detector                  98.8 Hz        10124 us
|   |__ 3) mc_hover_thrust_estimator       0.0 Hz            0 us
|   |__ 4) mc_pos_control                 50.0 Hz        19999 us
|   |__ 5) sensors                       199.5 Hz         5012 us
|   |__ 6) vehicle_acceleration          200.5 Hz         4987 us
|   |__ 7) vehicle_air_data               74.4 Hz        13443 us
|   |__ 8) vehicle_imu                   191.9 Hz         5212 us
|   \__ 9) vehicle_imu                   199.6 Hz         5009 us
|__ 7) wq:hp_default   
|   |__ 1) adc                           100.0 Hz        10000 us (10000 us)
|   |__ 2) battery_status                100.0 Hz        10000 us (10000 us)
|   |__ 3) pwm_out                        29.9 Hz        33471 us
|   |__ 4) rc_update                      44.3 Hz        22576 us
|   \__ 5) tone_alarm                     10.0 Hz        99752 us
\__ 8) wq:lp_default   
    |__ 1) load_mon                        2.0 Hz       499719 us (500000 us)
    \__ 2) send_event                     30.0 Hz        33332 us (33333 us)

sd_bench 8G Sans

INFO  [sd_bench]   Run  0:  1980.36 KB/s, max write time: 19 ms (= 210.53 KB/s), fsync: 10 ms
INFO  [sd_bench]   Run  1:  1974.17 KB/s, max write time: 17 ms (= 235.29 KB/s), fsync: 4 ms
INFO  [ecl/EKF] 9578728: EKF aligned, (baro hgt, IMU buf: 18, OBS buf: 14)
INFO  [sd_bench]   Run  2:  1963.01 KB/s, max write time: 18 ms (= 222.22 KB/s), fsync: 28 ms
INFO  [sd_bench]   Run  3:  1884.99 KB/s, max write time: 20 ms (= 200.00 KB/s), fsync: 22 ms
INFO  [sd_bench]   Run  4:  1895.87 KB/s, max write time: 19 ms (= 210.53 KB/s), fsync: 6 ms
INFO  [sd_bench]   Avg   :  1939.68 KB/s

Issues:
on 2.4.6 HW WARN [ecl/EKF] baro hgt timeout - reset to baro
hardfault_log fault does not reboot

davids5 · 2020-08-13T13:24:37Z

FMU V5 Holybro pixhawk4 SYS_AUTOSTART 4001
uorb top

update: 1s, num topics: 72
TOPIC NAME                     INST #SUB RATE #Q SIZE
actuator_armed                    0    5    2  1   24
actuator_controls_0               0    5  801  1   48
actuator_controls_3               0    2   45  1   48
actuator_outputs                  0    2   45  1   80
actuator_outputs                  1    1   85  1   80
adc_report                        0    1  100  1   96
battery_status                    0    5  100  1  112
battery_status                    1    5  100  1  112
cpuload                           0    3    2  1   16
estimator_sensor_bias             0    3  100  1   56
estimator_status                  0    3  100  1  288
input_rc                          0    3   45  1   72
manual_control_setpoint           0    4   45  1   72
multirotor_motor_limits           0    1   45  1   16
safety                            0    1    1  1   16
sensor_accel                      0    3  801  8   48
sensor_accel                      1    2 1014  8   48
sensor_baro                       0    1   75  1   32
sensor_combined                   0    2  200  1   48
sensor_gyro                       0    3  801  8   40
sensor_gyro                       1    2  998  8   40
sensor_mag                        0    2   89  1   48
sensor_mag                        1    2   89  1   48
system_power                      0    1  100  1   24
telemetry_status                  0    1    1  1  104
telemetry_status                  1    1    1  1  104
vehicle_acceleration              0    1  200  1   32
vehicle_air_data                  0    7   19  1   40
vehicle_angular_acceleration      0    1  801  1   32
vehicle_angular_velocity          0    4  801  1   32
vehicle_attitude                  0    5  200  1   48
vehicle_attitude_setpoint         0    3  200  1   64
vehicle_control_mode              0    8    2  1   32
vehicle_gps_position              0    4    5  1  104
vehicle_imu                       0    2  200  1   56
vehicle_imu                       1    2  199  1   56
vehicle_imu_status                0    3    9  1   56
vehicle_imu_status                1    3   10  1   56
vehicle_land_detected             0    8    1  1   24
vehicle_local_position            0   10  100  1  160
vehicle_magnetometer              0    2   89  1   24
vehicle_odometry                  0    1  100  1  256
vehicle_rates_setpoint            0    3  200  1   32

top

 PID COMMAND                   CPU(ms) CPU(%)  USED/STACK PRIO(BASE) STATE FD
   0 Idle Task                   41663 50.607   272/  512   0 (  0)  READY  3
   1 hpwork                          0  0.000   384/ 1284 249 (249)  w:sig  3
   2 lpwork                          0  0.000   352/ 1636  50 ( 50)  w:sig  3
   3 init                          491  0.000  2288/ 2948 100 (100)  w:sem  3
   4 wq:manager                      1  0.000   424/ 1276 255 (255)  w:sem  3
 199 commander                    1148  1.475  1472/ 3236 140 (140)  w:sig  6
  16 wq:lp_default                  88  0.086   984/ 1716 205 (205)  w:sem  3
  18 wq:hp_default                1348  1.562  1224/ 1916 240 (240)  w:sem  3
  20 dataman                        61  0.000   792/ 1220  90 ( 90)  w:sem  4
  26 wq:I2C1                       671  0.781   892/ 1492 246 (246)  w:sem  3
 134 wq:SPI1                     11551 13.020  1536/ 2356 253 (253)  PEND   3
 138 wq:I2C3                       532  0.607   892/ 1492 244 (244)  w:sem  3
 140 wq:SPI4                       277  0.347   892/ 2356 250 (250)  w:sem  3
 187 wq:nav_and_controllers       8666  9.982  5288/ 7220 241 (241)  w:sem  3
 188 wq:rate_ctrl                 3381  3.819  1176/ 1684 255 (255)  w:sem  3
 477 top                           183  2.604  1536/ 2052 239 (239)  RUN    3
 200 commander_low_prio              1  0.000   716/ 3020  50 ( 50)  w:sem  6
 209 mavlink_if0                  5364  6.163  1984/ 2596 100 (100)  READY  4
 246 gps                           137  0.173   976/ 1700 205 (205)  w:sem  4
 287 mavlink_if1                  1052  1.128  1888/ 2508 100 (100)  READY  4
 288 mavlink_rcv_if1               271  0.260  2552/ 4092 175 (175)  w:sem  4
 301 px4io                        3087  3.385  1160/ 1500 237 (237)  w:sem  4
 426 wq:attitude_ctrl              680  0.868  1296/ 1660 242 (242)  w:sem  3
 433 navigator                      65  0.086  1088/ 1788 105 (105)  w:sem  5
 471 mavlink_rcv_if0               379  0.260  2760/ 4092 175 (175)  w:sem  4

Processes: 25 total, 5 running, 20 sleeping, max FDs: 15
CPU usage: 46.61% tasks, 2.78% sched, 50.61% idle
DMA Memory: 5120 total, 1024 used 1536 peak
Uptime: 89.496s total, 41.663s idle

work_queue status

Work Queue: 9 threads                        RATE        INTERVAL
|__ 1) wq:rate_ctrl
|   |__ 1) mc_rate_control               801.6 Hz         1248 us
|   \__ 2) vehicle_angular_velocity      801.5 Hz         1248 us
|__ 2) wq:SPI1
|   |__ 1) bmi055                        997.5 Hz         1002 us
|   |__ 2) bmi055                       1013.4 Hz          987 us
|   \__ 3) icm20689                      799.8 Hz         1250 us
|__ 3) wq:SPI4
|   \__ 1) ms5611                         99.5 Hz        10046 us
|__ 4) wq:I2C1
|   |__ 1) ist8310                        89.0 Hz        11238 us
|   \__ 2) rgbled                         38.3 Hz        26142 us
|__ 5) wq:I2C3
|   \__ 1) ist8310                        89.1 Hz        11229 us
|__ 6) wq:attitude_ctrl
|   \__ 1) mc_att_control                200.4 Hz         4990 us
|__ 7) wq:nav_and_controllers
|   |__ 1) ekf2                          200.4 Hz         4990 us
|   |__ 2) land_detector                  89.5 Hz        11170 us
|   |__ 3) mc_hover_thrust_estimator       0.0 Hz            0 us
|   |__ 4) mc_pos_control                 50.0 Hz        19993 us
|   |__ 5) sensors                       200.2 Hz         4994 us
|   |__ 6) vehicle_acceleration          213.2 Hz         4690 us
|   |__ 7) vehicle_air_data               74.3 Hz        13454 us
|   |__ 8) vehicle_imu                   199.8 Hz         5006 us
|   \__ 9) vehicle_imu                   200.4 Hz         4990 us
|__ 8) wq:hp_default
|   |__ 1) adc                           100.0 Hz         9999 us (10000 us)
|   |__ 2) battery_status                100.0 Hz         9999 us (10000 us)
|   |__ 3) pwm_out                        86.9 Hz        11513 us
|   |__ 4) rc_update                      44.8 Hz        22322 us
|   |__ 5) safety_button                  30.3 Hz        32985 us (33000 us)
|   \__ 6) tone_alarm                     10.1 Hz        98809 us
\__ 9) wq:lp_default
    |__ 1) load_mon                        2.0 Hz       495972 us (500000 us)
    |__ 2) rgbled_pwm                     39.2 Hz        25541 us
    \__ 3) send_event                     30.0 Hz        33316 us (33333 us)

sd_bench 8G Sans

nsh> logger stop
INFO  [sd_bench] Using block size = 4096 bytes, sync=0
INFO  [sd_bench]
INFO  [sd_bench] Testing Sequential Write Speed...
INFO  [sd_bench]   Run  0:  2504.42 KB/s, max write time: 7 ms (= 571.43 KB/s), fsync: 6 ms
INFO  [sd_bench]   Run  1:  2486.88 KB/s, max write time: 9 ms (= 444.44 KB/s), fsync: 3 ms
INFO  [sd_bench]   Run  2:  2491.19 KB/s, max write time: 7 ms (= 571.43 KB/s), fsync: 3 ms
INFO  [sd_bench]   Run  3:  2505.77 KB/s, max write time: 7 ms (= 571.43 KB/s), fsync: 3 ms
INFO  [sd_bench]   Run  4:  2494.31 KB/s, max write time: 8 ms (= 500.00 KB/s), fsync: 3 ms
INFO  [sd_bench]   Avg   :  2496.52 KB/s

davids5 · 2020-08-13T15:18:20Z

@dagar - I was able to build the nuttx CI list locally. The mac flock should get us the rest of the way.

https://github.com/PX4/Firmware/runs/980783156

Can we get flock on today?

davids5 · 2020-08-13T16:22:56Z

holybro_durandal SYS_AUTOSTART 4001
uorb top

update: 1s, num topics: 70
TOPIC NAME                     INST #SUB RATE #Q SIZE
actuator_armed                    0    5    2  1   24
actuator_controls_0               0    6  802  1   48
adc_report                        0    1  101  1   96
battery_status                    0    6  101  1  112
battery_status                    1    6  101  1  112
commander_state                   0    1    2  1   16
cpuload                           0    4    2  1   16
ekf2_timestamps                   0    1  200  1   24
estimator_innovation_test_ratios  0    1  100  1  144
estimator_innovation_variances    0    1  100  1  144
estimator_innovations             0    1  100  1  144
estimator_sensor_bias             0    4  100  1   56
estimator_status                  0    4  100  1  288
logger_status                     0    1    1  1   40
rate_ctrl_status                  0    1  802  1   24
sensor_accel                      0    4  802  8   48
sensor_accel                      1    3  804  8   48
sensor_baro                       0    2   75  1   32
sensor_combined                   0    3  200  1   48
sensor_gyro                       0    4  802  8   40
sensor_gyro                       1    3 1001  8   40
sensor_mag                        0    3   89  1   48
sensor_preflight                  0    1  200  1   24
system_power                      0    2  101  1   24
telemetry_status                  0    2    1  1  104
telemetry_status                  1    2    1  1  104
vehicle_acceleration              0    1  200  1   32
vehicle_air_data                  0    8   19  1   40
vehicle_angular_acceleration      0    1  802  1   32
vehicle_angular_velocity          0    5  802  1   32
vehicle_attitude                  0    6  200  1   48
vehicle_attitude_setpoint         0    4  200  1   64
vehicle_control_mode              0    9    2  1   32
vehicle_imu                       0    3  200  1   56
vehicle_imu                       1    3  200  1   56
vehicle_imu_status                0    4    9  1   56
vehicle_imu_status                1    4   10  1   56
vehicle_land_detected             0    9    1  1   24
vehicle_local_position            0   11  100  1  160
vehicle_magnetometer              0    3   90  1   24
vehicle_odometry                  0    1  100  1  256
vehicle_rates_setpoint            0    4  200  1   32
vehicle_status                    0   17    2  1   56
vehicle_status_flags              0    1    2  1   40

top

 PID COMMAND                   CPU(ms) CPU(%)  USED/STACK PRIO(BASE) STATE FD
   0 Idle Task                  280964 75.731   272/  512   0 (  0)  READY  3
   1 hpwork                          0  0.000   352/ 1284 249 (249)  w:sig  3
   2 lpwork                          0  0.000   352/ 1636  50 ( 50)  w:sig  3
   3 init                          380  0.000  2176/ 2948 100 (100)  w:sem  3
   4 wq:manager                      0  0.000   424/ 1276 255 (255)  w:sem  3
 483 log_writer_file              1437  0.430   624/ 1188  60 ( 60)  w:sem  4
  17 dataman                        32  0.000   792/ 1220  90 ( 90)  w:sem  4
  19 wq:lp_default                  91  0.086   836/ 1716 205 (205)  w:sem  3
  31 wq:hp_default                2853  0.774  1080/ 1916 240 (240)  w:sem  3
 142 wq:SPI1                     27886  7.659  1608/ 2356 253 (253)  w:sem  3
 148 wq:I2C3                      1322  0.344   892/ 1492 244 (244)  w:sem  3
 150 wq:SPI4                       816  0.258   892/ 2356 250 (250)  w:sem  3
 208 wq:nav_and_controllers      16511  4.647  5288/ 7220 241 (241)  w:sem  3
 210 wq:rate_ctrl                 6501  1.807  1264/ 1684 255 (255)  w:sem  3
 218 commander                    2038  0.516  1440/ 3236 140 (140)  READY  6
 219 commander_low_prio              3  0.000   760/ 3020  50 ( 50)  w:sem  6
 226 mavlink_if0                  9944  2.753  1984/ 2596 100 (100)  READY  4
 265 gps                            95  0.086  1048/ 1700 205 (205)  w:sem  4
 300 mavlink_if1                  1954  0.516  1888/ 2508 100 (100)  READY  4
 301 mavlink_rcv_if1               562  0.172  2552/ 4092 175 (175)  w:sem  4
 314 px4io                         443  0.086  1096/ 1500 237 (237)  w:sem  4
 317 mavlink_rcv_if0               572  0.172  2552/ 4092 175 (175)  w:sem  4
 440 wq:attitude_ctrl             1441  0.430  1336/ 1660 242 (242)  w:sem  3
 453 navigator                     136  0.000  1032/ 1788 105 (105)  w:sem  5
 480 logger                       5258  1.462  2736/ 3660 230 (230)  w:sem  4
 494 top                            28  1.204  1544/ 2052 239 (239)  RUN    3

Processes: 26 total, 5 running, 21 sleeping, max FDs: 15
CPU usage: 23.41% tasks, 0.86% sched, 75.73% idle
DMA Memory: 5120 total, 1536 used 1536 peak
Uptime: 365.180s total, 280.964s idle

work_queue status

Work Queue: 8 threads                        RATE        INTERVAL
|__ 1) wq:rate_ctrl
|   |__ 1) mc_rate_control               801.6 Hz         1247 us
|   \__ 2) vehicle_angular_velocity      801.6 Hz         1247 us
|__ 2) wq:SPI1
|   |__ 1) bmi088                       1000.4 Hz         1000 us
|   |__ 2) bmi088                        804.7 Hz         1243 us
|   \__ 3) icm20689                      801.3 Hz         1248 us
|__ 3) wq:SPI4
|   \__ 1) ms5611                         99.7 Hz        10027 us
|__ 4) wq:I2C3
|   \__ 1) ist8310                        89.6 Hz        11159 us
|__ 5) wq:attitude_ctrl
|   \__ 1) mc_att_control                200.4 Hz         4990 us
|__ 6) wq:nav_and_controllers
|   |__ 1) ekf2                          200.4 Hz         4990 us
|   |__ 2) land_detector                  98.4 Hz        10159 us
|   |__ 3) mc_hover_thrust_estimator       0.0 Hz            0 us
|   |__ 4) mc_pos_control                 50.0 Hz        19998 us
|   |__ 5) sensors                       200.4 Hz         4991 us
|   |__ 6) vehicle_acceleration          202.3 Hz         4944 us
|   |__ 7) vehicle_air_data               74.8 Hz        13372 us
|   |__ 8) vehicle_imu                   200.1 Hz         4997 us
|   \__ 9) vehicle_imu                   200.4 Hz         4990 us
|__ 7) wq:hp_default
|   |__ 1) adc                           100.0 Hz        10000 us (10000 us)
|   |__ 2) battery_status                100.0 Hz        10000 us (10000 us)
|   |__ 3) pwm_out                         0.0 Hz    100237488 us
|   |__ 4) rc_update                       0.0 Hz            0 us
|   \__ 5) tone_alarm                      9.9 Hz       100521 us
\__ 8) wq:lp_default
    |__ 1) load_mon                        2.0 Hz       499896 us (500000 us)
    \__ 2) send_event                     30.0 Hz        33332 us (33333 us)

sd_bench 8G Sans

logger stop
INFO  [sd_bench] Using block size = 4096 bytes, sync=0
INFO  [sd_bench]
INFO  [sd_bench] Testing Sequential Write Speed...
INFO  [sd_bench]   Run  0:   640.37 KB/s, max write time: 13 ms (= 307.69 KB/s), fsync: 4 ms
INFO  [sd_bench]   Run  1:   640.43 KB/s, max write time: 11 ms (= 363.64 KB/s), fsync: 4 ms
INFO  [sd_bench]   Run  2:   638.67 KB/s, max write time: 10 ms (= 400.00 KB/s), fsync: 4 ms
INFO  [sd_bench]   Run  3:   642.43 KB/s, max write time: 10 ms (= 400.00 KB/s), fsync: 4 ms
INFO  [sd_bench]   Run  4:   638.62 KB/s, max write time: 10 ms (= 400.00 KB/s), fsync: 4 ms
INFO  [sd_bench]   Avg   :   640.10 KB/s

perf latency

bucket [us] : events
          1 : 46106
          2 : 720
          5 : 756
         10 : 56
         20 : 0
         50 : 0
        100 : 0
       1000 : 0

davids5 · 2020-08-13T19:48:39Z

px4_fmu-v5x SYS_AUTOSTART 4001
uorb top

update: 1s, num topics: 59
TOPIC NAME                    INST #SUB RATE #Q SIZE
actuator_armed                   0    3    2  1   24
actuator_controls_0              0    5  803  1   48
actuator_outputs                 0    3   45  1   80
battery_status                   0    6    2  1  112
battery_status                   1    6    2  1  112
commander_state                  0    1    2  1   16
cpuload                          0    4    2  1   16
multirotor_motor_limits          0    2   45  1   16
px4io_status                     0    1    1  1  128
rate_ctrl_status                 0    1  803  1   24
safety                           0    2    1  1   16
sensor_accel                     0    4  803  8   48
sensor_accel                     1    3  800  8   48
sensor_combined                  0    3  201  1   48
sensor_gyro                      0    4  803  8   40
sensor_gyro                      1    3 1002  8   40
sensor_mag                       0    3   28  1   48
sensor_mag                       1    3   89  1   48
sensor_preflight                 0    1  201  1   24
system_power                     0    2  100  1   24
telemetry_status                 0    2    1  1  104
telemetry_status                 1    2    1  1  104
vehicle_acceleration             0    1  201  1   32
vehicle_angular_acceleration     0    1  803  1   32
vehicle_angular_velocity         0    2  803  1   32
vehicle_control_mode             0    7    2  1   32
vehicle_gps_position             0    5    5  1  104
vehicle_imu                      0    3  201  1   56
vehicle_imu                      1    3  201  1   56
vehicle_imu_status               0    4    9  1   56
vehicle_imu_status               1    4    9  1   56
vehicle_land_detected            0    7    1  1   24
vehicle_magnetometer             0    3   89  1   24
vehicle_status                   0   14    2  1   56
vehicle_status_flags             0    1    2  1   40

top

 PID COMMAND                   CPU(ms) CPU(%)  USED/STACK PRIO(BASE) STATE FD
   0 Idle Task                  113707 60.927   252/  512   0 (  0)  READY  3
   1 hpwork                          0  0.000   400/ 1284 249 (249)  w:sig  3
   2 lpwork                          0  0.000   352/ 1636  50 ( 50)  w:sig  3
   3 init                          426  0.000  2272/ 2948 100 (100)  w:sem  3
   4 wq:manager                      1  0.000   408/ 1276 255 (255)  w:sem  3
   5 Telnet daemon                   0  0.000   544/ 2028 100 (100)  w:sem  0
 493 log_writer_file                 0  0.000   392/ 1188  60 ( 60)  w:sem  3
  17 wq:hp_default                 916  0.496  1192/ 1916 240 (240)  w:sem  3
  19 dataman                        65  0.000   792/ 1220  90 ( 90)  w:sem  4
  21 wq:lp_default                  63  0.000   836/ 1716 205 (205)  READY  3
  24 wq:I2C1                      1376  0.827  1032/ 1492 246 (246)  w:sem  3
 135 wq:I2C2                        56  0.082  1024/ 1492 245 (245)  w:sem  3
 137 wq:SPI1                      7900  4.552  1592/ 2356 253 (253)  w:sem  3
 141 wq:SPI3                     13240  7.533  1324/ 2356 251 (251)  w:sem  3
 144 wq:I2C4                       483  0.248   976/ 1492 243 (243)  w:sem  3
 187 wq:nav_and_controllers       8986  5.132  2056/ 7220 241 (241)  w:sem  3
 188 wq:rate_ctrl                 6828  3.890  1108/ 1684 255 (255)  w:sem  3
 198 commander                    1903  0.993  1280/ 3236 140 (140)  READY  6
 200 commander_low_prio              3  0.000   720/ 3020  50 ( 50)  w:sem  6
 210 mavlink_if0                  7571  4.304  1904/ 2948 100 (100)  w:sig  4
 267 gps                           276  0.165   976/ 1700 205 (205)  w:sem  4
 303 mavlink_if1                  2021  1.158  1904/ 2860 100 (100)  w:sig  4
 304 mavlink_rcv_if1               452  0.248  3520/ 5452 175 (175)  w:sem  4
 332 px4io                        5419  3.062  1052/ 1500 237 (237)  READY  4
 463 wq:attitude_ctrl                0  0.000   424/ 1660 242 (242)  w:sem  3
 475 navigator                      19  0.000  1032/ 1788 105 (105)  w:sem  5
 491 logger                        574  0.248  2192/ 3660 230 (230)  w:sem  3
 505 top                            74  3.062  1536/ 2052 239 (239)  RUN    3
 499 mavlink_rcv_if0               545  0.248  3520/ 5452 175 (175)  w:sem  4

Processes: 29 total, 5 running, 24 sleeping, max FDs: 15
CPU usage: 36.26% tasks, 2.81% sched, 60.93% idle
DMA Memory: 5120 total, 1024 used 1024 peak
Uptime: 178.316s total, 113.707s idle

work_queue status

Work Queue: 10 threads                        RATE        INTERVAL
|__ 1) wq:rate_ctrl
|   |__ 1) mc_rate_control               802.6 Hz         1246 us
|   \__ 2) vehicle_angular_velocity      802.6 Hz         1246 us
|__ 2) wq:SPI1
|   \__ 1) icm20602                      802.6 Hz         1246 us
|__ 3) wq:SPI3
|   |__ 1) bmi088                       1001.7 Hz          998 us
|   \__ 2) bmi088                        799.9 Hz         1250 us (1250 us)
|__ 4) wq:I2C1
|   |__ 1) ina226                          2.0 Hz       499315 us
|   |__ 2) ist8310                        89.1 Hz        11226 us
|   \__ 3) rgbled                         38.7 Hz        25817 us
|__ 5) wq:I2C2
|   \__ 1) ina226                          2.0 Hz       499310 us
|__ 6) wq:I2C4
|   \__ 1) bmm150                         28.5 Hz        35125 us
|__ 7) wq:attitude_ctrl
|   \__ 1) mc_att_control                  0.0 Hz            0 us
|__ 8) wq:nav_and_controllers
|   |__ 1) ekf2                          200.7 Hz         4984 us
|   |__ 2) land_detector                  20.0 Hz        50020 us
|   |__ 3) mc_hover_thrust_estimator       0.0 Hz            0 us
|   |__ 4) mc_pos_control                  0.0 Hz            0 us
|   |__ 5) sensors                       200.6 Hz         4985 us
|   |__ 6) vehicle_acceleration          204.4 Hz         4893 us
|   |__ 7) vehicle_imu                   200.4 Hz         4989 us
|   \__ 8) vehicle_imu                   200.7 Hz         4984 us
|__ 9) wq:hp_default
|   |__ 1) adc                           100.0 Hz        10000 us (10000 us)
|   |__ 2) pwm_out                         0.0 Hz     50566200 us
|   |__ 3) rc_update                       0.0 Hz            0 us
|   |__ 4) safety_button                  30.3 Hz        32996 us (33000 us)
|   \__ 5) tone_alarm                     10.1 Hz        99105 us
\__ 10) wq:lp_default
    |__ 1) load_mon                        2.0 Hz       499753 us (500000 us)
    \__ 2) send_event                     30.0 Hz        33328 us (33333 us)

sd_bench 8G Sans

nsh> logger stop
nsh>  sd_bench
INFO  [sd_bench] Using block size = 4096 bytes, sync=0
INFO  [sd_bench]
INFO  [sd_bench] Testing Sequential Write Speed...
INFO  [sd_bench]   Run  0:  2344.81 KB/s, max write time: 48 ms (=  83.33 KB/s), fsync: 3 ms
INFO  [sd_bench]   Run  1:  2788.61 KB/s, max write time: 102 ms (=  39.22 KB/s), fsync: 5 ms
INFO  [sd_bench]   Run  2:  2513.57 KB/s, max write time: 118 ms (=  33.90 KB/s), fsync: 12 ms
INFO  [sd_bench]   Run  3:  2602.06 KB/s, max write time: 7 ms (= 571.43 KB/s), fsync: 4 ms
INFO  [sd_bench]   Run  4:  3145.26 KB/s, max write time: 6 ms (= 666.67 KB/s), fsync: 3 ms
INFO  [sd_bench]   Avg   :  2678.71 KB/s

perf latency


bucket [us] : events
          1 : 19913
          2 : 637
          5 : 1425
         10 : 802
         20 : 122
         50 : 1
        100 : 0
       1000 : 1

dagar · 2020-08-14T18:16:57Z

For some reason cpu usage has increased significantly (first noticed on fmu-v4). I'll look into it.

master (px4_fmu-v4 SYS_AUTOSTART 4001)

PR (px4_fmu-v4 SYS_AUTOSTART 4001)

dagar · 2020-08-14T18:32:55Z

On fmu-v4 performance was still good (even better than master) up until 2 days ago (09ae5ac). http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-hardware/detail/pr-nuttx-9.1.0-/13/pipeline/624/

First bad build 178396a.

dagar · 2020-08-14T18:40:35Z

It was this NuttX change.

PX4/NuttX@ffa91e2...e9a0f5b

davids5 · 2020-09-10T20:35:45Z

@dagar From the NuttX perspective - this is ready to go when CI is happy. @MaEtUgR @TSC21 we should sync on what else is need from the build environment.

bkueng · 2020-09-11T07:42:23Z

 if [ -f $FRC ]
 then
-	sh $FRC
+	. $FRC


Can we use source everywhere for consistency?

I am glad you mentioned this, sure we can but do we want to inherit the env on this call out?

Yes the scripts currently assume that. Is that not the case on NuttX?

sh is now a new process. No parent ENV changes are possible. . or source are inclusions and can modify the parent.

MaEtUgR · 2020-09-11T09:27:06Z

I'm on my way to generate a new Cygwin Toolchain version with ARM GCC 9 (again) and checked flock is already present in the Cygwin basic Linux tools. What I do not understand yet is why all SITL tests fail during the startup script execution with this error:

/cygdrive/c/projects/firmware/build/px4_sitl_test/bin/px4-alias.sh: line 42: `source': is a special builtin
ERROR [px4] Startup script returned with return value: 512

Looks to me like it doesn't allow overriding the existing source shell command but I have to check locally.

MaEtUgR · 2020-09-11T10:16:43Z

@@ -28,6 +28,19 @@ sh() {
 	. "$(pwd)$script"
 }

+source() {


If I rename this to e.g. so() SITL tests all pass on Windows. I'm sure there's a reason to do this but Cygwin shell seems to not allow overriding the symbol source. Can you think of any workaround?

This comment hints already that overriding set and source is not a good idea:
https://github.com/PX4/Firmware/blob/2128679ab3e4ba888cc3f8b1cb9b00a398847cc5/platforms/posix/src/px4/common/px4-alias.sh_in#L7

The main difficulty is that both source and . require an absolute path.
If we need to avoid overriding altogether, we can add a prefix variable, $PWD on POSIX, and / on NuttX: . ${R}etc/init.d/rc.fw_defaults

@dagar argued we should clean all the invocations up and only uses sh hence the action listed in the title block after this comes in but maybe we need an interim fix:

Use $sh and assign to source on linux and . on nuttx?

Can we discuss it on the dev-call Wed?

I would do whatever works for now and once this settles I'll take a look to see if we can start isolating some of the scripts and not have to track setting and unsetting every variable in rcS.

MaEtUgR · 2020-09-11T10:17:40Z

@@ -89,6 +89,7 @@ function(px4_add_common_flags)
 		-Wno-missing-field-initializers
 		-Wno-missing-include-dirs # TODO: fix and enable
 		-Wno-unused-parameter
+


Suggested change

MaEtUgR · 2020-09-11T11:14:39Z

+ARCHINCLUDES += $(CINCPATH)
+ARCHXXINCLUDES += $(CINCPATH) $(CXXINCPATH)
+
+ifeq ($(CONFIG_CYGWIN_WINTOOL),y)


The include paths are all broken on Cygwin with this change. ifeq ($(CONFIG_CYGWIN_WINTOOL),y) doesn't detect Cygwin.

We'll probably need something like:

ifneq (, $(findstring CYGWIN, $(shell uname))) CONFIG_CYGWIN_WINTOOL = y

I started testing but don't have all paths properly converted for successful compilation yet.

I'm still looking for the correct way to deal with this. I have the detection and the TOPDIR conversion but still have to figure out the BOARD_DIR again. I feel it's all deja vu but forgot how it worked. I think last time I looked into it was 2018 #8737

dagar · 2020-09-17T01:27:35Z

As discussed on the dev call, let's get this in to get as much time as possible before the next release and continue to work through any remaining minor regressions with cygwin.

after NuttX 9.1.0+ upgrade #15139

after NuttX 9.1.0+ upgrade PX4#15139

davids5 mentioned this pull request Jun 18, 2020

Upgrade to nuttx 9.1.0- #15125

Closed

davids5 force-pushed the pr-nuttx-9.1.0- branch 3 times, most recently from 43b6745 to 1feeaa3 Compare June 19, 2020 22:51

davids5 requested a review from dagar June 19, 2020 22:59

davids5 force-pushed the pr-nuttx-9.1.0- branch 2 times, most recently from 139f50c to 8621b2a Compare June 23, 2020 13:24

dagar mentioned this pull request Jul 13, 2020

SPI_SETBITS of transferhword reads and writes data of more than 8 bits will cause an abnormal signal on the sck . #15073

Closed

davids5 force-pushed the pr-nuttx-9.1.0- branch 3 times, most recently from 9475bd5 to 0e90507 Compare August 11, 2020 21:17

davids5 force-pushed the pr-nuttx-9.1.0- branch from 0e90507 to 09ae5ac Compare August 12, 2020 21:13

davids5 force-pushed the pr-nuttx-9.1.0- branch from 09ae5ac to 178396a Compare August 13, 2020 14:14

davids5 force-pushed the pr-nuttx-9.1.0- branch from 178396a to b288de1 Compare August 14, 2020 12:47

dagar added enhancement labels Aug 14, 2020

davids5 added 7 commits September 10, 2020 13:31

px4_fmu-v5: Updates for NuttX 9.1.0+ External schedule_note

f47fe71

px4_fmu-v5x: Updates for NuttX 9.1.0+ External schedule_note

02b5ffb

px4_fmu-v5x: Updates for NuttX 9.1.0+ External schedule_note

cee661a

px4_fmu-v6x: Updates for NuttX 9.1.0+ External schedule_note

5702620

px4_fmu-v6x: Updates for NuttX 9.1.0+ External schedule_note

f87154f

spracing_h7extreme: Updates for NuttX 9.1.0+ External schedule_note

184de64

uvify_core: Updates for NuttX 9.1.0+ External schedule_note

ea4afb7

davids5 force-pushed the pr-nuttx-9.1.0- branch from 49be031 to ea4afb7 Compare September 10, 2020 20:32

bkueng reviewed Sep 11, 2020

View reviewed changes

MaEtUgR reviewed Sep 11, 2020

View reviewed changes

dagar approved these changes Sep 17, 2020

View reviewed changes

dagar merged commit c5f479d into master Sep 17, 2020

dagar deleted the pr-nuttx-9.1.0- branch September 17, 2020 01:32

This was referenced Sep 23, 2020

Regression with source command #15795

Closed

Windows build issue #15760

Closed

MaEtUgR mentioned this pull request Oct 13, 2020

PX4/JMAVSIM Build Error (return value 512) PX4/PX4-Devguide#1086

Closed

davids5 mentioned this pull request Oct 14, 2020

Remove source alias #15970

Merged

MaEtUgR added a commit that referenced this pull request Oct 23, 2020

Fix NuttX build under Cygwin

65f19d8

after NuttX 9.1.0+ upgrade #15139

MaEtUgR mentioned this pull request Oct 23, 2020

Fix NuttX build under Cygwin #16040

Merged

davids5 pushed a commit that referenced this pull request Oct 23, 2020

Fix NuttX build under Cygwin

4378254

after NuttX 9.1.0+ upgrade #15139

MaEtUgR mentioned this pull request Oct 26, 2020

Finally switch to ARM GCC 9, new Cygwin, new Ant, new openJDK PX4/PX4-windows-toolchain#23

Merged

mvacanti pushed a commit to mvacanti/PX4-Autopilot that referenced this pull request Nov 27, 2020

Fix NuttX build under Cygwin

c4c43cd

after NuttX 9.1.0+ upgrade PX4#15139

Conversation

davids5 commented Jun 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davids5 commented Jun 19, 2020

Uh oh!

dagar commented Jun 19, 2020

Uh oh!

davids5 commented Jun 23, 2020

Uh oh!

davids5 commented Aug 12, 2020

Uh oh!

davids5 commented Aug 12, 2020

Uh oh!

davids5 commented Aug 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davids5 commented Aug 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davids5 commented Aug 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davids5 commented Aug 13, 2020

Uh oh!

davids5 commented Aug 13, 2020

Uh oh!

davids5 commented Aug 13, 2020

Uh oh!

dagar commented Aug 14, 2020

master (px4_fmu-v4 SYS_AUTOSTART 4001)

PR (px4_fmu-v4 SYS_AUTOSTART 4001)

Uh oh!

dagar commented Aug 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dagar commented Aug 14, 2020

Uh oh!

davids5 commented Sep 10, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaEtUgR commented Sep 11, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaEtUgR Sep 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dagar commented Sep 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

davids5 commented Jun 18, 2020 •

edited

Loading

davids5 commented Aug 13, 2020 •

edited

Loading

davids5 commented Aug 13, 2020 •

edited

Loading

davids5 commented Aug 13, 2020 •

edited

Loading

dagar commented Aug 14, 2020 •

edited

Loading

MaEtUgR Sep 11, 2020 •

edited

Loading