On some Android phones, you can set USB connection preferences in Settings > Wireless and network > USB utilities, where you have options for selecting the USB mode when the cable is plugged in. If you select None, then the device gets recognized, the USB interface put up on the GNU PC, and you can ssh in.

However, on the Samsung Galaxy Note, there is no such option, and from the USB utilities dialog you can only select to put the phone in mass storage mode. As a result the USB internface is not brought up by the GNU PC and you can't connect.

The only way I've found to connect so far is:

• Download the GNU Android SDK;
• Unpack the archive;
• Run android-sdk-linux/tools/android;
• In the Tools section, install Android SDK Platform-tools;
• Put android-sdk-linux/platform-tools/adb in your $PATH;

Now on the phone:

• Install QuickSSHd from the Android Market;
• Start QuickSSHd and turn the service on;

To connect to the phone now:

• Plug in the USB cable;
• On the GNU PC:

  # adb forward tcp:22 tcp:2222

This will redirect the local tcp port 22 to the port 2222 on the phone, on which the SSH server will be running now, served by QuickSSHd.

To connect from the GNU PC then:

References:
http://forum.xda-developers.com/showpost.php?p=20104928&postcount=71