Recently in the Ubuntu-UK LoCo team IRC channel we had a new user arrive with a problem on their Ubuntu computer. Nothing new there, it happens every day, but the way the problem was solved was slightly different than the way we usually do it. I’m looking for feedback on this.
Emma told us that Ubuntu worked fine but at 800×600, and she couldn’t get it to run at a higher resolution such as 1024×768 on her TV/Monitor. It had previously worked at a higher resolution on another family members display. Over a period of about two hours late one evening we had an exchange of log files from Emma, and commands and config files from various people in the channel back to Emma.
I don’t want to go into massive detail about the problems, but in summary there seemed to be a mixture of versions of the binary nVidia driver installed, coupled with a display which didn’t correctly report its capabilities.
We collectively crafted a configuration file which should have fixed things, but due to it being late and done in a hurry a mistake slipped in, so it didn’t work. The next day Emma returned to our little channel and we continued with the investigation.
After a further hour of back and forth with Emma, I decided that the easiest (for her) and quickest (for me) way to solve it would be to remotely control her computer. I suggested this and Emma agreed this was a viable option.
The method I used to remotely control was to provide Emma with the following:-
1) A method to switch from the GUI to the console via the keyboard (and back):-
CTRL+ALT+F1 and CTRL+ALT+F7.
2) A command she could execute which would log her onto my computer, and allow me remote control (SSH) access to her machine. Something like this:-
ssh emma@myhomepc.dyndns.org -R22222:localhost:22
3) A method for her to kill the connection should she wish to.
4) My (@ubuntu.com) email address and my mobile phone number.
5) When finished, a complete rundown of what I did to her machine to make it work.
She had another machine which she could use to get back on irc whilst I was bouncing her machine, so I could ask questions like ‘Is it at the right resolution now?’. Of course sometimes I could figure that out from the logs, but it’s always nice to have remote hands/eyes to check things.
It took about an hour (in between me cooking lunch for my children) for me to resolve the issue. Overall it was probably no more than 30 minutes work for one expert user, but delays are of course introduced when passing log files and commands back and forth which don’t happen when the ‘expert’ has direct access to the machine. Overall I think it went pretty well.
However, this process raises a few issues, and I wondered what other people in the Linux community do about this kind of thing – these are in no particular order:-
Side band communication
I needed to continue to be able to communicate with Emma during the diagnostic process, and it was lucky that she had another computer (also running Ubuntu) so that she could use to talk to me on IRC.
Many new users may only have one computer. How would I communicate with the person whilst their computer is in an unstable state? She had my phone number and email address.
Remote desktop (VNC) not an option?
During the problem diagnosis and resolution I had to restart X a few times. So I figured that traditional Remote Desktop or even Gitso would not be useful here as the connection would drop, closing the support connection. It’s also no good starting an SSH tunnel from inside a graphical terminal on the desktop because it would drop as soon as I restarted X to test changes.
We need an easier way for users to be able to open a channel for remote control via ssh. Once the remote user requiring support can paste in the command it’s not difficult for me to tunnel over it, but it’s hardly elegant. I’m beginning to think we need a console based equivalent to Gitso which allows people to setup, monitor and close an SSH session for someone else to come in over. Would also be nice if there was a little chat window in it so we could talk to eachother.
Perhaps something using screen -x with a split window?
Trusting the other person
Emma hadn’t spoken to me before and didn’t know me from Adam. I could have been anyone, and could have done anything to her computer including installing keyloggers, accessing personal data, sniffing the network and so on. Of course I _didn’t_ do any of those things. But it was possible.
I did ask up front whether there was any data on the machine, not because I expected her to have to do a reinstall, but because I felt more comfortable touching a machine that I knew didn’t have much ‘user baggage’.
How can a new user know who to trust?
Trust goes both ways
In order that I could SSH to her machine over a tunnel I asked her to SSH to one of my machines. I clearly have to trust that she isn’t going to do nafarious things to my computer, just as she does with me on hers.
Is there some way of opening a remote SSH tunnel such that the user opening the connection can’t do anything on the remote end? Maybe it’s good that they can.. Mutually Assured Distruction 
Remote user root/sudo
In order to effectively resolve the issue quickly I asked for Emmas password to allow me to use sudo. I could of course have poked about on her machine as another user, but using hers was again easier.
Perhaps we need to be able to use a guest ‘support’ account for remote support? A user whose every move is logged somewhere they can’t delete it, and has no rights to use sudo?
I’d be interested in hearing from other Ubuntu support people how they go about helping ‘strangers’, and ways in which I can improve the process I went through to allow Emma to access my machine, and me hers.
15 Comments
Were you the first one to suggest that you remote in or did she suggest it?
Did she understand the risks associated with letting you or anyone else remote in? Did your good deed just build an expectation that other people she meets and asks for help will be honest. You might have just reinforced a pattern of behavior which will lead to bigger problems. Next time a system is behaving poorly, is she going to jump in on irc and get remote access to he first person who offers to help?
1) I never offer to remote into a system of a person I do not know. This is more for their protection than my own. I will not unwittingly encourage them to open themselves up to malicious social engineering attacks.
2) When they offer to give me remote access. I make it a point to explain to the person that there’s no way for them to be sure that I’m not malicious and that they are taking a risk..before I do anything. Once its clear they understand the risks, they seldom want me to remote in.
3) What you really want to try to do is set up a screen session on their machine that you enter into. Via screen they can watch everything you do. That helps. Yes screen has multi-user support for local users. Ideally you want the multi-user screen session on their computer not yours. That way as you do commands on their system. I’ll leave it as an exercise on how to force your ssh connection to the remote machine to dump you into a running screen session.
I suggested it and her response was along the lines of ‘I was about to suggest the same thing’.
11:17 < popey> one suggestion I have is that I would be willing to remotely control your computer to figure out the issue and hopefully fix it
11:17 < popey> now, this means I'd have access to your computer, if you're not happy about that then thats fine
11:17 < emma> yeah no problem i was about to offer the option to you
11:17 < popey> but if you're happy I can (at your request) logon remotely, and hit it with a (virtual) stick
Maybe I should setup a boilerplate piece of text to explain the kind of warnings and caveats that you outlined Jef.
I’m certain this kind of thing could be a simple application. It could be made to _only_ allow itself to be run from a tty on the local machine, so it’s known that the person doing it is really the local ‘owner’.
Here’s my point… sometimes being helpful requires you to think like someone who isn’t. A social engineering attack to gain remote access is going to look a lot like your irc interaction. You shouldn’t trust anyone who overtly suggests that they be granted remote access to your computer…full stop. It does not matter if its the easiest way to diagnose the problem..its also the easiest way for to create a much bigger problem.
Hell man, even if you are well respected in channel…the people in channel can not vouch that the person writing as popey in the conversation is actually you. If you keep all the communication in a public channel…then the help is peer reviewable and someone can raise a red flag if you suggest something obviously malicious.
But…if you are serious about an application that will automate something like a multi-user screen session so your actions can be tracked…and logged. you’ll want to get familiar with how rssh works or how safekeep works for backups. ssh does allow you to force individual authorized keys to run specific commands on connect by editting the authorized keys file. Safekeep has a utility that automates this. You could easily design an application for someone like emma to use which would configure an authorized key line to drop you into a restricted logged backed multi-user screen session.
You might even talk to Kirkland about a dedicated screen profiles usage case for remote help.
Interesting post. In some cases directly working on the remote system is really the easiest way to help, but esp. the trust thing is a problem IMO. Your idea about a dedicated support account sounds great, though. Maybe it could be confined to a shared Screen session, so the user always sees what the supporter does and can enter sudo password if necessary. This would be a really cool addition.
Isn’t there a way of limiting SSH keys on the server end to only run certain remote commands (for example /bin/false)? Maybe a solution involving the user downloading a passphraseless private key from your website to allow automatic authentication to one of your own boxes with a false shell? The only problem I can think of with this approach is that there’s a few steps (get the private key, make an SSH config file that tells SSH to use that (passphraseless) key for connection to your box, and also set up the port forwarding).
It’s probably scriptable somehow but then you have the problem of the user having to download and trust your script since they may not be fluent in shell script
It could also be integrated with Launchpad to grab your key from there, of course that’s only useful for Ubuntu.
I agree that it needs to be a package that is either pre-installed or easily installed.
But that would be retrieving *your* key to *their* machine allowing you to access their machine? I was thinking more for what you said above, about a remote user SSHing to your machine, and how do you trust them not to be nefarious (or even just somehow clumsy!) as you can’t rely on the network setup/NAT allowing you directly into their machine.
I was thinking that with a passphraseless key and the correct .ssh_config then the “ssh support@popeysmachine -R2222:localhost:22″ would be a) easier, as the command line options would be reduced meaning less typing as it’s well publicised that you should never blindly copy and paste things from IRC/MSN and b) secure-r at your end as that particular SSH key/account could be limited to doing nothing apart from holding the connection open.
Maybe some sort of “ubuntu-remote-support” package or PPA with keys, configs and bash scripts to connect to a handful of remote volunteers who could be listed on the Ubuntu website or wiki as “trusted” helpers?
/me shrugs, it’s early, I’m rambling now
Console jabber-client+telepathy+tubes+remote control?
Recommended console based jabber client – remember I can’t use empathy or pidgin due to the GUI going away. Well, I could run up a vnc server but that seems like over kill.
At least, the tubes+telepathy option would efficiently solve the initial problem of connecting to the user.
Did port forwarding need to be setup on her router to allow you in? Or was her computer directly connected to the internet?
Getting them to configure port forwarding may be a tricky thing to setup, but gisto or yuuguu.com could be used for that GUI based activity I guess.
No. I already had a port open at my end which I use when away from home to SSH into my home machine. She just used that. Reverse SSH Tunnel is an awesome thing. The person wanting help just opens a connection as above (the critical bit is the -R parameter) which opens a port at the remote (to her) end back to the local (to her) end. So from my perspective I
ssh -p 22222 emma@localhostand via the magic of SSH I end up on her machine.Gitso uses exactly the same method (I believe) so only the helper needs a port open, not the person being helped.
I don’t have a magic answer for ssh. With the webbook project I was doing remote support, but assuming the GUI was running I ran a listening VNC viewer, the remote user just put in some commands to throw their desktop at me. Something similar for ssh would appear to be buildable based on this kind of idea. You kind of want most of the effort on the supporter side in terms of ports and firewalls, you have to assume the user is behind NAT. In this case you want them to start a session and throw that at you and give you control. You don’t want to type in any password, or do anything invisible to the user (they are supposed to be learning after all).
As mentioned earlier you can change the shell of the user they log in as on your computer. I think if you set it as something like /bin/cat it keeps the connection open, but they don’t have access to bash.
Can we get a wiki or something up to brainstorm this
?
My ideas are:
- helper program has ability to essentially chroot the ssh session to known areas (e.g., for yours, you needed /etc which tends to not contain much personal information).
- complete transparency of logs available… all commands recorded etc.
- change various warning banners, motd banners etc., so if the user is dumped to a console, it tells them how to fix the problem! E.g., PLEASE PRESS CTRL + ALT+ F6 or something
- the helper service should be a background daemon that can survive reboots, X-restarts, and login & log outs.
- on login, the helper service should offer to resume the session, which would provide a web-based chat client to talk to the support person + a window to see the commands the support person is typing
- provide the option for the user being helped to require manual approval of each command typed by the support person
- the latter could be linked to community feedback to give a rating on the likely “dangerousness” of a command? Though this might encourage some vicious gaming to combine non-dangerous commands into dangerous ones
- ability to tell a remote support user is who they claim they are, and a way to leave feedback in a community area on how well they helped you (reputation if you will)
One Trackback
[...] http://popey.com/blog/2009/08/26/trust-in-remote-support/ [...]