latency-test, latency-histogram: add realtime status bar

[grandixximo]

What

latency-test and latency-histogram now show a status bar along the bottom with three fields: the running realtime type (or No realtime), the CPU frequency governor, and isolcpus. A field is coloured red only when it flags a condition that makes the measured latency unrepresentative, so normal states stay in the theme colour. On a real RT machine it reads e.g. Preempt RT / performance / isolcpus=2,3.

Why

In #4044 a run-in-place build was used without sudo make setuid. rtapi_app ran unprivileged, the latency numbers blew up, and it was mistaken for a code regression. A visible realtime indicator surfaces that immediately. The governor and isolcpus fields cover the other two common reasons a latency measurement is misleading.

How

The realtime type now comes authoritatively from rtapi itself: realtime verify returns the type that rtapi_app reports for the running backend (Preempt RT, RTAI, Xenomai, ..., or No realtime), instead of each tool re-sniffing kernel signals. The governor is read from /sys and isolcpus from /proc/cmdline. latency-test renders the bar as a pyvcp footer; latency-histogram as a Tk status bar at the bottom. The histogram's earlier modal no realtime popup is replaced by this persistent bar (the console warning is kept for non-X runs).

Docs: install/latency-test.adoc gains a status-bar section explaining each field and linking the fixes (realtime kernel, setuid/setcap, isolcpus), plus a CPU frequency governor tuning note.

rtapi_app change (needs RT/C++ review)

To make the type available without sniffing, this branch carries @hdiethelm's change to rtapi_app (src/rtapi/uspace_rtapi_main.cc): the check_rt socket result is extended to return a string alongside the int status, and the realtime-type enum is mapped to a human-readable name. I added a follow-up commit tidying it: a short-read check in recv_result(), a frame-size underflow guard in recv_result()/recv_args(), an exhaustive type-name switch, and typo fixes. The two commits are kept separate so the tidy delta reads on its own.

Follow-ups (separate PRs)

• A realtime indicator in the main GUIs (axis, gmoccapy, qtdragon, touchy), showing just the realtime field; cross-GUI placement is Feature: Properly warn if no realtime kernel is active #4118.
• A headless --quiet/--seconds mode for latency-test so scripts can capture latency (suggested by @rodw-au).

Depends on #4132 (merged). Closes #4044.

[BsAtHome]

These tests are moot when you run on a non-RT kernel (like I do in dev). I'm not sure the noise is really necessary in that case.

[grandixximo]

silenced on non-rt

[rodw-au]

Great. Every little bit helps. Reduces user frustration and more importantly less developer time wasted on spurious issues.

[hdiethelm]

Hmm, in #4044 in the images in the background you clearly see:

So the new warning will not help that much. If desired, you can add it in the place where Note: Using POSIX non-realtime is printed, so at least i is shown always, not only in these test tools.

[hdiethelm]

Connected to this:
#4118

A general way for GUI's to show "You don't have realtime" warnings that you can not overlook would help the most. When milling, I start linuxcnc with the link. So no console. If I accidentally start the wrong kernel, bad luck.

[BsAtHome]

On a production system, you may want to warn all the time when this is amiss. Maybe something with a background color turning from gray into gray-red tinted and do something similar consistently in all GUIs.

For dev builds you don't really want this because you want to see what the operator sees while you are working on stuff. I'd go for an opt-in choice by setting a value in the INI file. Maybe something like a boolean [DISPLAY]VISUAL_WARN_NONRT, defaulting to false. Then add the entry commented out in our configs and add a choice in pnconf and friends.

[hdiethelm]

I was just checking the code. @grandixximo You already added a better warning in the C code in your nonroot patch, so this was probably not even applied in the screenshot. Now there is a double warning in the console. The note from C++ and the Warning from this PR:

With latency-histogram there is also a pop-up. With latency-test, this doesn't work. And the most important app, linuxcnc, also just shows noting if you don't start it in a console.

Anyone has a good idea to check easily and globally for real time capability?

There is already a function rtapi_is_realtime(). This is not 100% reliable, if harden_rt() fails, it will return true, even if it runs in SCHED_OTHER. But this can be fixed.

rtapi_is_realtime() is also linked to userspace apps but there it will not work, it checks if the userspace app has realtime... ;-)

I could add a halcmd that checks for realtime. Or a pin that is true when all is ok, false otherwise.

This could then be used in all gui's for an opt-in or opt-out warning. But I am not that deep into all these various gui's and how they communicate with the RT part.

[hdiethelm]

On a production system, you may want to warn all the time when this is amiss. Maybe something with a background color turning from gray into gray-red tinted and do something similar consistently in all GUIs.

For dev builds you don't really want this because you want to see what the operator sees while you are working on stuff. I'd go for an opt-in choice by setting a value in the INI file. Maybe something like a boolean [DISPLAY]VISUAL_WARN_NONRT, defaulting to false. Then add the entry commented out in our configs and add a choice in pnconf and friends.

Might be an option that is default on when you deploy an default off in rip-mode? But this is annoying to test.

Otherwise, I would tend for default on, dev's will manage it better to switch it off than users will fight not knowing that they don't have real time enabled. Instead of in ini, might be an environment variable LINUXCNC_NO_RT_WARN. Dev's can set it on their dev machine in .profile if they are annoyed and it works for all test configs.

BTW, just brainstorming options.

[hdiethelm]

Just a POC, if you think a halcmd getrt (or better name) would help I can create a PR. Was easy.
You can use that everywhere and it will return 1 if failed / 0 if good.

../bin/halcmd getrt ; echo Return value $?
<commandline>:0: exit value: 1
<commandline>:0: No realtime available
Return value 1

make setuid

../bin/halcmd getrt ; echo Return value $?
Realtime available
Return value 0

You find it on my fork:
https://github.com/hdiethelm/linuxcnc-fork/tree/halcmd_getrt
hdiethelm/linuxcnc-fork@master...hdiethelm:linuxcnc-fork:halcmd_getrt

[hdiethelm]

Meanwhile, I found also something that looks like it is exposed to the python code:

linuxcnc/src/hal/halmodule.cc

Line 2382 in 888cb94

PyModule_AddIntConstant(m, "is_rt", rtapi_is_realtime());

But this is broken: #4129

[grandixximo]

Thanks both, this is more useful than my original per-tool heuristic.

I have pivoted the PR to use @hdiethelm's halcmd getrt as the single source of truth instead of hand-rolling a setuid-bit / getcap probe in bash and tcl. Both scripts now just run halcmd getrt and warn only when it reports No realtime available. An rtai/non-uspace build, an older halcmd without getrt, or a working realtime setup all stay silent, so the check rides on the authoritative rtapi_is_realtime() path rather than guessing from file permissions. This also drops the weaker logic @hdiethelm rightly flagged.

A few things I would like your input on, since they touch the broader direction in #4118:

1. Console double-warning. rtapi already prints Note: Using POSIX non-realtime at the source (uspace_posix.cc). For a console tool like latency-test that note is arguably enough, and a second line from the script is the duplication you saw. I am inclined to keep the script warning only for its actionable hint (the make setuid / make setcap pointer) and let the GUI popup be the real value-add in latency-histogram. Happy to drop the latency-test console line entirely if you would rather the C note be the only console source.

2. getrt invocation/cleanup. Since getrt goes through hal_systemv and a HAL init, calling it standalone before the test seems to bring up an rtapi instance. @hdiethelm, where do you intend callers to invoke it, and does it need a halrun -U afterward so it does not collide with the session the tool then starts? I did not want to bake in a cleanup that could disturb a running setup.

3. Dev opt-out policy. I wired a LINUXCNC_NO_RT_WARN env opt-out per @hdiethelm's suggestion, which keeps @BsAtHome's dev boxes quiet without a per-kernel heuristic. If the consensus in Feature: Properly warn if no realtime kernel is active #4118 lands on an INI key like [DISPLAY]VISUAL_WARN_NONRT instead, I will switch to that. The env var is easy to set once for all test configs, which is why I started there.

This now depends on the getrt command landing. @hdiethelm, if you open that as its own PR I will rebase on top and reference it.

[BsAtHome]

Instantiating a HAL memory segment on invocation may be problematic. It will surely confuse because you have to remember to call halrun -U afterwards. That is not a good design.

Opt-out policies are generally designed to force you to do a thing, even if you do not want to. That is why they should be avoided.

[hdiethelm]

Instantiating a HAL memory segment on invocation may be problematic. It will surely confuse because you have to remember to call halrun -U afterwards. That is not a good design.

The way halcmd getrt runs this in the brackground for uspace is by executing rtapi_app getrt. Now there are two possibility's:

• rtapi_app is already running (for example you start latency_test in an other terminal): The command is executed and the result returned.
• rtapi_app is not yet running: master starts, runs the command and exits again due to instance_count==0. So nothing stays behind. No halrun -U needed. However, this is kind of a low-likelyness race condition: If you manage to break realtime somehow between halcmd getrt and halrun lat.hal, then no error is reported.

You see that in the following test where I added a message when rtapi_app exits:

halcmd loadrt and2
#Note: Using POSIX realtime
halcmd getrt
#Realtime available
halcmd getrt
#Realtime available
pgrep rtapi_app
#3799
halrun -U
#exit master

vs

halcmd getrt
#Note: Using POSIX realtime
#exit master
#Realtime available
halcmd getrt
#Note: Using POSIX realtime
#exit master
#Realtime available
pgrep rtapi_app
#no process running

I don't see any big downside in doing it like that. But i also do not 100% like starting up rtapi_app just to exit right away. Better would be running it with or after halrun in the script. Or might be an approach using a signal / parameter.

Of course, also RTAI / doc and so on needs to be checked / updated before I will call that ready.

Better ideas are welcome. But I prefer using a check executing the same code path for realtime checks always instead the variant before where you then have most likely diverging real time checks spread in all possible apps.

[grandixximo]

@hdiethelm I have restructured to avoid the standalone HAL instantiation @BsAtHome flagged:

• latency-histogram now calls halcmd getrt from inside its own running session (right after hal start), so it attaches to the realtime already up rather than spinning up a segment that needs a separate halrun -U.
• latency-test drops its script-side check entirely and relies on the existing Note: Using POSIX non-realtime from rtapi, which already lands on the same console. That also removes the double-warning you saw.
• Dropped the LINUXCNC_NO_RT_WARN opt-out per @BsAtHome; the dev-suppression policy can be decided in Feature: Properly warn if no realtime kernel is active #4118 rather than baked in here.

That keeps the scripts honest, but @BsAtHome's deeper point lands on getrt itself: do_getrt_cmd goes through hal_systemv + a HAL init, so any standalone caller instantiates a segment. Is it worth making getrt probe rtapi_is_realtime() without a full hal_init, so a GUI can ask "is realtime available" cheaply before starting anything? That would let every GUI (including linuxcnc started from a launcher, the #4118 case) query it without session side effects. If you think that is the right shape, this PR can depend on that and I will rebase on top once your getrt lands as its own PR.

[hdiethelm]

@grandixximo Nice. I have to test it.

I created a PR, feel free to rebase:
https://github.com/LinuxCNC/linuxcnc/pull/4132/changes

@BsAtHome
Yes, rtapi_app has the annoying behavior that it always creates this memory segment and initializes RT. Even if you exit right away after. However, it looks like this doesn't hurt anything, you can start any app after rtapi_app getrt or other commands that do not increase the instance counter or do anything else than initializing this segment and exiting afterwards.

Any hint's what should be done in this case? Before my pr rtapi_app rework pr, even rtapi_app exit initialized a memory segment when it was not running. ;-)

[grandixximo]

Crossed posts, @hdiethelm. Good, your "run it with/after halrun in the script" is exactly what I did for latency-histogram (getrt after hal start, inside the session), so no stray rtapi_app and it also avoids the break-in-between race you noted. For latency-test I leaned on the existing rtapi note rather than a second getrt call; happy to switch it to getrt inside its HAL flow if you would rather every tool go through the one path. I will rebase on your getrt PR once it is up with the RTAI/doc bits.

[hdiethelm]

Hmm, just an idea:
Something like this for the test scripts? I find this huge popup's a bit annoying.

@BsAtHome
What do you think about this for all GUI apps? TBD how to inhibit but there will be a way.
4803bb1
Somehow gmocappy doesn't show this error. I guess there is a bug that startup errors are not shown?

[BsAtHome]

A (forced) popup is the equivalent of slapping someone in the face.

The error message added to the GUI is actually not an error. Not running RT on a production system may be considered an error.

If you look closely in AXIS' status bar you see "Kein Werkzeug". That is also the place where you want to warn the user. Add a status bar field that is obvious (light red background) yet not invasive.

[hdiethelm]

You have a point. I also get annoyed of all this popup's when using the good old microslop... :-D

Now on the status bar: Good idea. How to do that? I am already somewhat deep in the C code, so I can add any needed support there but for GUI's, someone else has to take over.

@grandixximo Can you do this based on whatever from the hal? halcmd / parameter / pin is easy to add for me.

[grandixximo]

@hdiethelm thanks, I will rebase this onto #4132 once it settles. @BsAtHome agreed the popup is too much; I will drop the tk_messageBox and warn non-invasively in the test tools instead.

On the cross-GUI status bar, that is the right shape and I am happy to do the AXIS side (a status-bar field with a light-red background, like the existing tool slot) once @hdiethelm's "realtime ok" signal from #4132 exists. The question is where it lives.

My preference is to keep #4107 narrow: rebased on getrt, popup gone, scoped to the latency tools. It can ship as soon as #4132 lands. The GUI-wide warning cannot be written until the HAL pin exists and touches AXIS, gmoccapy and qtvcp, so I would do it as a separate PR tracking #4118 rather than make this small change wait on the slowest part.

That said, if you would rather have one PR own the whole intent, I am fine rescoping #4107 to the GUI-wide warning and retitling it; it just becomes larger and slower. Either way works for me. Which do you prefer?

[grandixximo]

@hdiethelm to answer your "halcmd / parameter / pin" question directly: a bool pin is best for the GUIs. AXIS, gmoccapy and qtvcp already monitor HAL pins, so they can reflect realtime state live in the status bar without polling a command. I would steer away from a param given those are heading for deprecation, and a halcmd is the least convenient since a GUI would have to shell out to poll it.

[BsAtHome]

I think we first have to agree on the proper conceptual design of how to detect in the different scenarios and what to do with it.

[grandixximo]

@BsAtHome agreed, let me put a concrete proposal on the table.

Detection: one source of truth. @hdiethelm's rtapi_is_realtime() path, exposed as a single bool HAL pin ("realtime ok"). Every app reads the same signal, so there is no divergent per-app logic, which was the original concern.

What to do with it: per UI, not one mechanism. The right surface differs by app, so each owns its own rendering rather than forcing a single widget everywhere:

• Console tools (latency-test): the existing Note: Using POSIX non-realtime already covers them.
• GUIs (AXIS, gmoccapy, qtvcp): an in-window, non-invasive indicator. Obvious but not a slap, e.g. a light-red status-bar field, no forced popups.

One thing to rule out: coloring the window title bar / decoration is not reliable. Plenty of setups have no title bar at all (fullscreen/kiosk panels, some Wayland/WM configs), so the signal has to live inside the app window, not in the chrome.

Still open (defer): production-vs-dev suppression policy. That can ride with #4118 once the pin and the per-UI rendering exist.

Does that match how you see the scenarios?

[hdiethelm]

Sounds like a plan. I will create new PR with a signal. Then we can test how this feels and continue from there.

I can do that tomorrow, right now I have other things to do.

About the title bar: The idea was to only use this for the two test apps. If this is cumbersome, might be just modify the text that is already displayed in them.

@grandixximo Can you mark this PR as a draft until we are done?

Sorry about the for- and back. If i dont have a good solution yet, this is often my way of brainstoming. Try things and discard until it is good. Hope this is ok for you.

[BsAtHome]

Detection: one source of truth. @hdiethelm's rtapi_is_realtime() path, exposed as a single bool HAL pin ("realtime ok"). Every app reads the same signal, so there is no divergent per-app logic, which was the original concern.

That is only partly satisfactory because for this realtime needs to be running.

You want to know in advance whether your system will be capable of running RT without starting any of it. Then, when you are running, you want to know from various applications what the actual status is by using generic API call or/and HAL pin.

[grandixximo]

Hope this is ok for you.

No problem, we brain storm it and come up with something that sticks.

[rodw-au]

Seeing you are working on this, what would be a useful feature to add would be a command line switch that ran for a specific --time --quiet(ly) and print the latency results to the console so scripts could report latency. I've wanted to do this recently and had a lot of trouble finding a way to do this. cyclictest allows you to do this but it's still not easy and not within the LinuxCNC ecosystem.

[hdiethelm]

Looks nice in the footer! Visible but not a slap in the face.

I did not know how to do this, mainly the reason why I used the title bar to add some ideas.

Is this also possible in axis / gmoccapy?

[grandixximo]

Seeing you are working on this, what would be a useful feature to add would be a command line switch...

Good idea, belongs as a --quiet/--seconds mode of latency-test, please open an issue, it's a different PR.

Is this also possible in axis / gmoccapy?

Yes, but On the main GUIs I'd show just the realtime field (RT-<type> / no realtime); the governor and isolcpus stay in latency-test / latency-histogram.

Placement, reusing each GUI's existing status area:

• Axis: the existing status footer (the "No Tool" / position row).
• QtDragon: its status bar.
• Gmoccapy: there's room near the clock. (I don't really see anywhere else, and I don't know if we are allowed to make it bigger with a footer, to discuss in Feature: Properly warn if no realtime kernel is active #4118?)
• Touchy: it already has a status bar; it goes there.

Just the one field via lcnc_realtime.verify() plus the type for the label.

Keep it as its own PR, separate from this one, it's the cross-GUI convention #4118 should settle. This PR lands as the latency-tools footer.

[hdiethelm]

It looks good with the footer.

However, I don't like the now 2x copy paste real time check code. This is guaranteed to brake soon and be not consistent. For example, for xenomai, rtapi_app still needs setuid, so it is already not consistent when I use setcap on a xenomai kernel:

Now I wold rather not show the type than doing it this way.

Right now, you can check the effectively running type by using python type = hal.get_realtime_type() but this is a bit cumbersome, it works only when rtapi_app is running, so you would have to do that after halrun lat.hal in a separate thread.

Alternatives:

• realtime verify: I could print the realtime type on stdout. So I can capture it in python and forward it.
• Using non-standard return values: Probably a bad idea. I would have to use 1...9 for the types and 0 for error
• The hard way: Properly separate rtapi_app in master / client and create a client library, so library calls can be used to communicate with the rtapi_app master.

[hdiethelm]

I tried out the variant passing the text from rtapi_app trough the socket up to realtime verify and python:
https://github.com/hdiethelm/linuxcnc-fork/tree/fix/latency-setuid-warning-4044
hdiethelm@4d80249

It works but just quickly coded together, needs some doc and tidy up. Do you think this will do the job?

I changed only latency-test to use this. TCL is unreadable for me, would take me hours to do it... ;-)

[andypugh]

Happy to drop the latency-test console line entirely if you would rather the C note be the only console source.

I don't think this is worthwhile, as that line is also useful to tell you which realtime system you are using.
So I think it is good to say which RT you are on, and additionally warn that it is not RT.

3. Dev opt-out policy. ... The env var is easy to set once for all test configs, which is why I started there.

I think I like the env var. the vast majority of users need to be warned. Those who habitually test-run the code on a non-RT system are unlikely to mind clicking-away a warning.

[grandixximo]

@hdiethelm @BsAtHome a heads-up on the rtapi_app part of this PR.

To give the status bar an authoritative realtime type, I folded @hdiethelm's check_rt-returns-the-type-string POC (4d802490da) into this PR. That expands the scope here beyond the GUI status bar into rtapi_app (src/rtapi/uspace_rtapi_main.cc). hdiethelm built his refinement on top of my work rather than as its own PR, so merging and expanding scope felt like the natural move to avoid a rebase-and-wait. I do not mind splitting it back out into a separate PR if that is preferred, @BsAtHome especially if you would rather review the rtapi change on its own.

What the change does: the check_rt socket result is extended to return a type string alongside the int status, and the realtime-type enum is mapped to a human-readable name (Preempt RT, RTAI, Xenomai, ..., or No realtime). The serialization mirrors the existing send_args/recv_args framing.

I added one tidy commit on top (40300eedd2), kept separate so the delta reads on its own:

• recv_result() now checks the second recv_data() for a short read, like recv_args() already does.
• Guard recv_result()/recv_args() against a frame size below its own length prefix, which would underflow buff_size.
• Dropped a dead out.resize() overwritten on the next line.
• Moved the type-to-name map into realtime_type_name() with an exhaustive switch and no default, so -Wswitch flags a new enum value.
• Typo fixes (to big) and a stale send_args comment in send_result().

@hdiethelm could you confirm you are happy with the tidy and drop the POC marker from your commit if it is good to land? It is your RT domain, so I did not touch the protocol design itself.

@BsAtHome a review of the wire-format and C++ would be appreciated since that is your area. Happy to adjust to whatever framing convention you prefer.

[grandixximo]

@andypugh both your points are covered by where this landed, though the shape changed since that comment.

On showing which RT: the latency-test / latency-histogram footer now shows the realtime type directly (Preempt RT, RTAI, Xenomai, ..., or a red No realtime), sourced authoritatively from realtime verify rather than sniffed. So it says which RT you are on and flags when it is absent, in-window, which also covers the menu-launched case where the console note is not visible. The rtapi Note: Using POSIX ... console line is untouched.

On the env-var opt-out: I dropped LINUXCNC_NO_RT_WARN after Bertho's point. The footer is non-invasive (a status-bar field, not a popup), so there is nothing to click away and no opt-out is needed here. If a more assertive GUI-wide warning ever does need suppression, that policy belongs in #4118 alongside the cross-GUI work, not baked into this small PR.

[hdiethelm]

@hdiethelm could you confirm you are happy with the tidy and drop the POC marker from your commit if it is good to land? It is your RT domain, so I did not touch the protocol design itself.

I will look into it again let's say until tomorrow. Thanks for pulling my commit in and correcting some parts. I have to check how I can allow you to push to a repo under my github account. We are working at the same thing much lately and having two branches working at the same issue just generates unnecessary friction.

[grandixximo]

I could have revised the original commit and pushed under your name from my end, as the author could have stayed, but I really did not want to do that, I'd rather keep it as it happened for now, and I can squash later, or possibly separate the concerns in two PR, waiting for Bertho on the final judgement on that. The friction is more about waiting your PRs to land. Pulling in your commit and commit on top is zero friction for me. I can push PRs to your fork, I just decided to go this route instead, why does it create friction on your end? You can just pull merge and commit on top, no? I can merge your changes here when you are done, I don't see much friction.

[hdiethelm]

I could have revised the original commit and pushed under your name from my end, as the author could have stayed, but I really did not want to do that, I'd rather keep it as it happened for now, and I can squash later, or possibly separate the concerns in two PR, waiting for Bertho on the final judgement on that. The friction is more about waiting your PRs to land. Pulling in your commit and commit on top is zero friction for me. I can push PRs to your fork, I just decided to go this route instead, why does it create friction on your end? You can just pull merge and commit on top, no? I can merge your changes here when you are done, I don't see much friction.

Ok, fine for me. Waiting for other PR's is for sure friction. You can also squash your and my commit together and just mark me in the comment.

So we just go this route, I push a commit to my version of this branch and then you pull it in and squash it at the end.

Most is fine, except some not updated comments. I will quickly update them and the you can pull in my commit. You can remove the POC at the squash phase.

[hdiethelm]

Cleanup from my side: hdiethelm@b7f4618
This will close my comments.

[grandixximo]

@hdiethelm done on all your points, branch force-pushed.

• Pulled in your cleanup commit (Pass realtime string: Cleanup), so the protocol doc-comment and the send_result size calc are your versions.
• Dropped the duplicate non-realtime warning in latency-histogram: check_rt_privileges was repeating what rtapi_app already prints on the console, and the footer surfaces it persistently in the GUI, so the script no longer says it a second time. That closes your scripts/latency-histogram comment.
• Squashed the branch into two commits, as we agreed:
  • rtapi_app: return the realtime type name from check_rt (your POC + cleanup + my tidy), authored by you, POC marker gone, with me as co-author for the tidy fixups.
  • latency-test, latency-histogram: add realtime status bar (the GUI side), mine.

The tidy fixups folded into your commit are the short-read check in recv_result(), the frame-size underflow guard in recv_result()/recv_args(), the exhaustive type-name switch, and the typo fixes; shout if you would rather any of those drop.

@BsAtHome the rtapi_app wire-format change is now a single self-contained commit (c93e8e857b) if you want to look at it on its own.