Top
Best
New

Posted by piccirello 1 day ago

SSH Secret Menu(twitter.com)
https://xcancel.com/rebane2001/status/2031037389347406054
230 points | 92 comments
0xbadcafebee 11 hours ago|
Find the HIDDEN SECRETS that THEY DON'T WANT YOU TO KNOW!

    $ man ssh
longislandguido 5 hours ago||
Many Linux man pages have the thoroughness of a fortune cookie, so I can understand the skepticism.

Jackpot if they're just a pointer to an 'info' page.

gcr 6 minutes ago|||
Most people just discount man pages as unreadable and don’t even try to understand them.

Case in point: the jq man page is incredible and everyone I know instead runs off to google or stackoverflow or Claude to answer simple questions

Almondsetat 32 minutes ago||||
The real jackpot is if they're the same as the --help command
halapro 1 hour ago||||
I was never able to properly parse large man pages, I'm so happy that llms can now prepare half a usable command without spending an hour reading a time without a single usage example.
jeremyscanvic 57 minutes ago||
What I usually do when I have to read large man pages like bash(1) is I read them as PDFs:

man -Tpdf bash | zathura -

Replace zathura with any PDF viewer reading from stdin or just save the PDF. Hope that can be useful to someone!

aa-jv 2 hours ago||||
But its also true that many, many man pages have extremely valuable information that no enterprising hacker should overlook, too ..
darkwater 3 hours ago|||
Do we still have those? I think it was common in late '90s, due to GNU trying to get `info` gain moment but nowadays?
skywhopper 2 hours ago||
Mostly just the bash builtins have (had) this problem.
imwally 9 hours ago|||
I blew a few minds a week ago when I told my younger coworkers about man pages. I hit ‘em with the `man man` and one dude was like “whoa!”
AndrewDavis 5 hours ago|||
Further blow their minds by showing them `apropos` https://manpages.debian.org/testing/man-db/apropos.1.en.html
SEJeff 17 minutes ago|||
man -k, apropos, but less to type
riffraff 4 hours ago|||
I remember my first days with Linux/Unix and being taught about man and apropos and info. But somehow I could never find anything useful via apropos!

With many years of insight, I think I probably never updated the database.

bombcar 9 hours ago||||
Reminds me of the old canard of 'man woman'
Aardwolf 1 hour ago||||
man, cat, lynx... We're all just creatures in a vast universe
aa-jv 2 hours ago||||
$ man -k <whatever>
vlod 24 minutes ago|||
I still struggle with the fact that I was (or delusional) an effective C programmer "back in the day" (before google etc) and all we had was "man" to look up std/x11 system calls.

Now I am dismayed with juniors who can't even be bothered to use google (or llms) to look up stuff on their first hiccup.

#include <old-man-shouting-at-clouds>

SEJeff 16 minutes ago||
That’s not a builtin, so wouldn’t it be:

   #include <old-man-shouting-at-clouds.h>
obezyian 3 minutes ago||
And it's kinda old, so maybe

  #define _POSIX_C_SOURCE 1
  #include <old-man-shouting-at-clouds.h>
antonvs 3 hours ago|||
Who doesn’t want you to know? Well, obviously, the man. That’s why you type man ssh, you’re forcing the man to tell you what he knows.
saguntum 9 hours ago|||
man ssh_config is even more interesting and hidden

ProxyCommand is fun

project2501a 3 hours ago||
Honest question, why is ProxyCommand `fun`? What do I get out of ProxyCommand that i do not get out of setting the correct order for ProxyJump and doing an ssh finalhost -- domy --bidding?
pmontra 3 hours ago|||
ProxyJump is a newer functionality. There used to be only ProxyCommand. ProxyJump is a shortcut for the usual way to use ProxyCommand to connect through a bastion host but ProxyCommand is more flexible. For example with ProxyCommand you can run any command to connect to the remote host. ProxyJump only connects over ssh. I think I replaced all my ProxyCommand with ProxyJump because I don't need much else than the normal use case.
m132 2 hours ago||||
You can get a lot more out of ProxyCommand. For example, you can run SSH over non-IP protocols, such as serial, Bluetooth RFCOMM for embedded boards, or vsock for virtual machines without networking set up at all. The latter is built into and setup up automatically by systemd:

https://www.freedesktop.org/software/systemd/man/257/systemd...

pritambaral 3 hours ago||||
ProxyCommand allows you to use any command to setup a connection. Not necessarily an ssh command, like ProxyJump. It can be any command, as long as it receives on stdin and produces on stdout, it can act like a TCP connection.

ProxyJump is a special case of `ProxyCommand ssh -p <port> <user>@<host>`. Can't replace the `ssh` in there when using ProxyJump.

brigandish 2 hours ago|||
I use ProxyCommand to run spipe tunnels for SSH.
smoyer 10 hours ago|||
HN as a tabloid ... After all enquiring minds want to know!
paulddraper 6 hours ago|||
It’s been a while since I’ve heard RTFM
senectus1 10 hours ago|||
hahaha made me laugh, thanks :-D
thaumasiotes 3 hours ago||
How's that supposed to help? The ssh man page is about as close as you can get to a Platonic example of "uninformative pretend-documentation".
hrmtst93837 2 hours ago||
If the ssh man page were any lazier it would just be a list of arguments with no context at all, so I get the frustation. The tricky bit is that much of the "secret menu" knowledge about ssh only lives in blog posts and random issue comments, which makes the doc gap feel intentional at times. If you want real examples or usage patterns you pretty much have to scrape Stack Overflow or crawl through dotfiles in public repos hoping someone else already fought the same fight.
andrewflnr 6 hours ago||
I've been using the ~. shortcut for a while, but somehow escaped learning about the help menu.

Another neat thing I noticed while playing with it just now: there's an option to enter ~ twice to send a literal ~, but usually you don't have to do this when typing something like 'ls ~' in a regular session. Not only does the ~ have to be the first character on a line to start an escape sequence, but typing on a line, backspacing all the way to the start and then typing ~ also sends a literal tilde. It only triggers the escape sequence if the ~ is the chronologically first character after a newline (or first in the session), which is an unlikely thing to type into a shell in a normal session. Good choice of UI, both the character and the state machine.

bandie91 1 hour ago||
> typing on a line, backspacing all the way to the start and then typing ~ also sends a literal tilde

for the younger readers, yes, because in terminal echo mode, "backspacing" does not clear your terminal line buffer, those characters backspaced are already sent on the line. if you ever seen a misconfigured terminal, it hints what's going on, like:

user@host$ ls ~/^?^?^?^?^?~/a.out

^? is backspace's control char.

that is ssh watches what you type, not what is on the screen (terminal).

computerfriend 1 hour ago||
The drawback is that if you think your session is hanging and want to bail with ~., you have to press enter, which might actually make it to the server and execute something.
bandie91 1 hour ago||
many get used to Ctrl-U, Return, ~, period keystroke sequence for this.
piccirello 11 hours ago||
I've been using SSH for ~15 years and never knew about these escape sequences. I'm eagerly awaiting my next hung session so that I can test `~.`. It's much nicer than my current approach of having to close that terminal window.
tomxor 9 hours ago||
If hung SSH connections are common it's likely due to CGNAT which use aggressively low TCP timeouts. e.g. I've found all UK mobile carriers set their TCP timeout as low as 5 minutes. The "default" is supposed to be 2 hours, you could literally sleep your computer, zero packets, and an SSH connection would continue to work an hour later, and generally speaking this is still true unless CGNAT is in the way.

If you are interested there are a few ways you can fix this:

Easiest is to use a VPN, because the VPN's exit node becomes the effective NAT they usually have normal TCP timeouts due to being less resource constrained. Another nice benefit of this method is you can move between physical networks and your connection doesn't die... If you use Tailscale then you already have this in a more direct way.

Another is to tune the tcp_keepalive kernel parameters. Lowering the keepalive timeout to be less than the CGNAT timeout will cause keepalive probes to prevent CGNAT from dropping the connection even while your SSH connection is technically idle. For Linux I pop these into /etc/sysctl.d/z.conf, I have no idea for Windows or Mac:

  # Keepalive frequently to survive CGNAT
  net.ipv4.tcp_keepalive_time   = 240 
  net.ipv4.tcp_keepalive_intvl  = 60
  net.ipv4.tcp_keepalive_probes = 120
This is really a misuse of these settings, they are supposed to be for checking TCP connections are still alive and clearing them up from the local routing table. Instead the idea is to exploit the probes by sending them more frequently to force idle connections to stay alive in a CGNAT environment (dont worry the probes are tiny and still very infrequent).

_time=240 will send a probe after 4 mins of idle connection instead of the default 2 hours, undercutting the CGNAT timeout. _intvl=60 and _probes=120 mean it will send 120 probes 60 seconds apart (2 hours worth) before considering the connection dead. This will keep it alive for at least 2 hours, but also allows us to have the best of both worlds so that under a nice NAT it keeps the old behaviour, e.g if I temporarily lose my network the SSH connection is still valid after 2 hours, but under CGNAT it will at least not drop the connection after 5 mins so long as I keep my computer on and don't lose the network.

There are also some SSH client keepalive settings but I'm less familiar with them.

vbezhenar 3 hours ago|||

    Host *
        ServerAliveInterval 25
snvzz 6 hours ago||||
Note this is only an issue if not using IPv6.

CGNAT is for access to legacy IPv4 only.

rnhmjoj 5 hours ago|||
Well, for different reasons, but you have similar issues with IPv6 as well. If your client uses temporary addresses (most likely since they're enabled by default on most OS), OpenSSH will pick one of them over the stable address and when they're rotated the connection breaks.

For some reason, OpenSSH devs refuse to fix this issue, so I have to patch it myself:

    --- a/sshconnect.c
    +++ b/sshconnect.c
    @@ -26,6 +26,7 @@
     #include <net/if.h>
     #include <netinet/in.h>
     #include <arpa/inet.h>
    +#include <linux/ipv6.h>
     
     #include <ctype.h>
     #include <errno.h>
    @@ -370,6 +371,11 @@ ssh_create_socket(struct addrinfo *ai)
      if (options.ip_qos_interactive != INT_MAX)
        set_sock_tos(sock, options.ip_qos_interactive);
     
    + if (ai->ai_family == AF_INET6 && options.bind_address == NULL) {
    +  int val = IPV6_PREFER_SRC_PUBLIC;
    +  setsockopt(sock, IPPROTO_IPV6, IPV6_ADDR_PREFERENCES, &val, sizeof(val));
    + }
    +
      /* Bind the socket to an alternative local IP address */
      if (options.bind_address == NULL && options.bind_interface == NULL)
        return sock;
gspr 3 hours ago||
Interesting! Is there anywhere a discussion around their refusal to include your fix?
rnhmjoj 44 minutes ago||
See this, for example: https://groups.google.com/g/opensshunixdev/c/FVv_bK16ADM/m/R...

It boilds down to using a Linux-specific API, though it's really BSD that is lacking support for a standard (RFC 5014).

dsl 5 hours ago|||
This is a very common misconception. The issue is not IPv4 or CGNAT, it's stateful middleboxes... of which IPv6 has plenty.

The largest IPv6 deployments in the world are mobile carriers, which are full of stateful firewalls, DPI, and mid-path translation. The difference is that when connections drop it gets blamed on the wireless rather than the network infrastructure.

Also, fun fact: net.ipv4.tcp_keepalive_* applies to IPv6 too. The "ipv4" is just a naming artifact.

anthk 4 hours ago||||
Check Mosh. It supports these kind of cuts and it will reconnect seamlessly. It will use far less bandwidth too. I successfully tried it with a 2.7 KBPS connection.
iberator 4 hours ago|||
putty is sending packets for network up since like forever
lathiat 11 hours ago|||
Have been using that weekly since probably 20 years. Will change your life :)

My other favourite is I very often SSH with -v to figure out why the connection is hanging, you rapidly figure out if DNS is failing, the TCP connection doesn't open, it does open but no traffic flows at all or it opens and SSH negotiation starts but never finishes. You can learn a lot just from this about what is wrong.

Izkata 8 hours ago||
Also helps with auth failures, I've used it several times with co-workers who can't figure out why their ssh key isn't working. It lists the keys out and some extra information.
sirfz 10 hours ago|||
You can even chain them if you have deep ssh connections (i.e. ssh from one instance to another). I think it would be ~~. to terminate the 2nd hop.

Edit: it's already explained in the OP

tdeck 5 hours ago|||
You don't need to actually open the menu either. Just hit enter, tilde, ., enter.
aa-jv 2 hours ago|||
I last used this menu about 20 years ago when a dialup modem was the only way to roll, and have pretty much forgotten about it since the days of always-on direct to the desktop TCP/IP ..
fragmede 8 hours ago|||
Just ssh to funky.nondeterministic.computer to test it out!
wolvoleo 11 hours ago|||
I use that every day but it's the only one I know by heart lol
TacticalCoder 10 hours ago|||
> It's much nicer than my current approach of having to close that terminal window.

You can also just kill the ssh process (say from another terminal). That way you get to keep your terminal window. And this works with everything "blocking" your terminal, not just ssh.

shmerl 11 hours ago||
I've been using ~. on hung ssh connections for a while.
vzaliva 9 hours ago||
I've used ~. for a long time but did not know about others. I know, should have read man page.

Anyway, if you try it from shell prompt it is likely will not work as pressing ENTER shows the next prompt. Try `cat` followed by ENTER and then ~?

ralferoo 1 hour ago||
Same with me, I'll still instinctively go for ~. when a connection has hung / dropped (usually because of a NAT via a rebooted firewall), but never even considered how ~ doesn't normally cause an issue. Never knew it had to be immediately following a newline. Also never knew about the other options, ~^Z in particular looks useful.

I wonder if anyone still remembers the ctrl-[ sequence in telnet. I think I only ever used the quit command in that though.

BenjiWiebe 9 hours ago||
It'll still work. OpenSSH doesn't care about output (for ~ stuff), only input, so if you type <enter>~. it will close the connection.
krautsauer 7 hours ago|||
Does not for me, not even with busybox sh and no funky escape codes in PS1 at all. It does with cat or yes running, so just something being output is not the problem… Hm.
vzaliva 7 hours ago|||
It does not. open ssh linux to mac, typing ~ just types it on fish shell prompt. It works after`cat` followed by ENTER
ploxiln 5 hours ago|||
Just type <enter> without cat, your shell will show you another prompt, and the ssh escape command will also work.
gear54rus 49 minutes ago||
No they are correct, fish seems to intercept this or something like that. Only works with cat.
dnet 2 hours ago|||
In newer versions, it's disabled by default and you have to do something like this to enable in ~/.ssh/config:

    Host *
    EnableEscapeCommandline yes
nh2 28 minutes ago||
`EnableEscapeCommandline` only controls the <Enter>~C commandline.

The reason that is disabled in current OpenSSH by default is OpenBSD `pledge` support:

https://security.stackexchange.com/questions/280793/what-att...

On my Linux,

    cat<Enter>~.
closes the connection as expected, and no ~ is shown in the terminal.
wrs 8 hours ago||
Those aren’t “secret”, they’re obviously borrowed from rsh — oh that’s right, I’m old.
aidenn0 7 hours ago||
And these days you need to pass "-O" to scp to have it behave like rcp.
aa-jv 2 hours ago||
Laughs in uucp!bangpath.
pram 11 hours ago||
Yep the menu is handy for ssh tunneling. Maybe not a lot of people doing that these days though with stuff like dev tunnels and Tailscale.
MayeulC 10 hours ago|
I typically just create a "new" connection in a separate tab when I want to add tunneling.

I put new in quotes because I use another little-known feature, "ControlMaster". Multiplexes multiple connections into one, it makes making " new" sessions instant (can also be configured to persist a bit after disconnecting). Also useful for tab-completing remote paths. It does not prompt for authentication again, though. And it's a bit annoying when the connection hands (can be solved with ssh -o close, IIRC).

NitpickLawyer 5 hours ago|||
> I use another little-known feature, "ControlMaster". Multiplexes multiple connections into one, it makes making " new" sessions instant

Is this what secureCRT used as well? I remember this being all the rage back when I used windows, and it allowed this spawn new session by reusing the main one.

ghrl 6 hours ago||||
I'm using that as well but had issues with tunneling where it creates the tunnel in the background and terminates and so you might not know the random port it assigned or I couldn't figure out how to un-tunnel it and tunnel again to the same port. Just bypassed the control master then.
nyanchovy 9 hours ago|||
TIL; thanks, that's interesting (and somehow escaped my 20+ years of using ssh)! As usual the gold is in the comments :-)
blueflow 11 hours ago||
As secret as the ssh manual.
bigstrat2003 9 hours ago|
"secret" not in the sense that it's hidden, but that most people won't know about it. Because approximately nobody actually reads man pages in their entirety, they just get in to find out how a specific flag works and then get out.
utopiah 3 hours ago||
My gosh... I've spent decades closing a connection by killing the terminal.

~.

juancn 10 hours ago||
It's like Ctrl + ] on telnet.

The good old times!

elric 3 hours ago|
Unlike CTRL ], at least ~. doesn't require that I press two modifiers at the same time ... CTRL ALTGR $. Because people who define those kinds of shortcuts never consider how they might work on non-QWERTY layouts.
teddyh 35 minutes ago||
Try Ctrl-5.
xnf 2 hours ago|
Sometimes things feel so simple that i dont even read the manual. cool
More comments...