bpence wrote:

I think I have at least a partial answer. The host can change the keyboard modes by sending escape sequences. This is why the behavior seems to change without you doing anything. The server is doing it. To determine why and when the server is doing it may require more analysis. However, to turn off the behavior in Absolute, try disabling the 'options->properties->vtoptions-VTKEYPAD' option. By disabling VTKEYPAD, the kaypad and arrow keys should retain their functions regardless of what the server tries to tell them.

Now normally, this option only affects the keypad, including the arrow keys on the keypad. However, it appears that your arrow keys are behaving as if they were keypad arrow keys, if this makes any sense. Is there anything unusual about your keyboard? Is it a standard 101 key keyboard? Laptop keyboard?

Brian

It is a standard 101/102 (I haven't counted 🙂 ) key keyboard on a Dell Inspiron desktop (floor, actually) computer, nothing unusual about the keyboard. In fact, I have been using this hardware for probably a year and a half, AT all that time, with no problems until just before I opened this thread. With almost minuscule exceptions, I access this shell account only with AT. My ISP (Panix in New York City) has some in-house newsgroups and does provide support, so if necessary/appropriate we can ask them for information. (Please let me know what to ask.)

In the meantime, I am changing the option as you suggest. We will see what happens, and I will keep you posted. As always, thanks.

Paul Bartlett

I unchecked (changed) the option as you suggested, and I have had no further problems with the directional keys as such. However, a new problem has arisen.

Whether we English-speakers like it or not, the blunt, irrevocable, obnoxious as it may seem to some, fact is that not all the world speaks English, and not all the world uses the US-ASCII character set. This is just a blunt factor of reality.

I subscribe to various mailing lists and newsgroups, through my shell netBSD account which I AT/SSH2 into, and some of those posts deal with languages other than English, including posts in the ISO 8859-1 / Latin-1 character set. I use the joe editor (runs under Unix/netBSD/Linux), which allows entering characters with the high bit set ("upper 128-range characters" ) in Latin-1 by means of almost trivially simple control sequences. But if I uncheck the "VT keypad" option as was suggested, I can no longer enter the "upper 128-range characters" with my preferred editor, which I must have.

So It seems like AT gives me two options: renounce the upper-128 range of characters and abandon English, which is simply unacceptable, or renounce the use of the directional keys.

Any assistance will be welcome.

Paul Bartlett

Paul,

Thanks for the feedback.

How, exactly, are you entering these characters?

AbsoluteTelnet has support for many international character sets as well as support for keyboard entry via Microsoft input methods. I believe the correct way to enter these characters is by going to control panel regional settings and adding the correct keyboard for the language you're trying to type. On the task bar, you should then see the language bar to switch between keyboards (you can even assign hot-keys to do this).

Then, on Absolute's Options->Properties->Appearance tab, select ISO8859-1 as your translation. THen, anything you type using any keyboard will be converted to ISO8859-1 (if it can) on the way to the host. To be even more international, you can do the same thing with UTF-8, but you need to make sure that your email client, news reader, and editor all understand what character set the terminal wants to see. Typically, this can be done by correctly setting the LANG environment variable.

Let me know if any of this helps.

As for the character set, I most commonly keep it set in AT as ISO 8859-1, as that is the most common charset (after USASCII) appearing in the posts of the mailing lists (and occasionally newsgroups) I use. Sometimes there are posts in Win 1252 (I think that is the number), but they are rarely a problem.

Unless I have been doing something drastically wrong for all the years I have been using AT, when I connect with the server (over the years it has been SunOS, Linux, and now netBSD depending on the ISP), at that point my AT (SSH2) session is really acting as if it were just a dumb terminal directly connected by a physical link to the server, with a few parameters set here and there. I might as well have a copper wire twisted pair connection to New York City (I am in Virginia) with an old fashioned Wyse or DEC VT terminal.

I mostly (not quite entirely) use Pine for email and news once I am logged into the server in character mode (no graphics here). It is provided by the ISP, and what they have is what I get. (Yes, I know, the University of Washington has abandoned Pine and Alpine.) When I want to compose a message, the edit space opens up with the joe editor, which is the SunOS/Linux/netBSD editor I have been using for years. I sometimes also activate joe from the command line. There is no appreciable difference as such when I do so with opening it by Pine.

In order to enter high-bit-set characters, I type Ctrl- (that's Control-backslash) and a key from the lower-128 -- I have a table -- and that creates the upper-128 character. joe as such does not know and does not care what character actually *appears* on the monitor, as that is a function of the display. joe is merely entering uninterpreted bytes. It has no knowledge of any Microsoft keyboard jiggering around. Apart from the Meta (Ctrl-+key) (Control-backslash plus key) function that I have described, joe knows nothing of Microsoft keyboard entries. It is a 7-bit ASCII editor with the ability to enter 8-bit bytes using the Ctrl- (Meta-) mechanism, leaving any and all interpretation of those 8-bit bytes elsewhere. (Incidentally, joe does not understand DBCS, so I cannot enter DBCS UTF-8 characters. That is sometimes why I somoetimes have to go outside AT entirely.)

Unfortunately, except when I am at the command line, Pine gets into the picture, as in its .pinerc file there is a character-set parameter, which seems to have something to do with things. On rare occasions I have reset that parameter to UTF-8 and from the command line set some (tcsh) environment variables. Then I go into AT and change the option for UTF-8. That is an awful lot of fumbling around just to change character sets for one or a few messages, and I know from sad experience that I have to go through *all* those steps.

At this point, there is no realistic possibility of doing anything with Pine, as it is provided by the ISP. Unfortunately, for those of us dealing with non-English languages at times, it is not feasible to control what charsets foreign posters use: some use ISO Latin-1, some Win 1252, and some UTF-8. It is just unpredictable.

Sadly, the combination of AT and Pine seems not to allow message-by-message adaptations, as some clients do (sometimes I use Opera), rather than changing everything globally. As I say, I have been a happy AT user for years, but this non-ASCII language issue makes me wonder if I should just junk the shell-account plus command-line-and-Pine interface and go to something else.

Thanks again.

Paul Bartlett

Edit: apparently this board does not like to display Control-backslash literally.

The *right* way to do this is to configure AbsoluteTelnet to use Utf8 and Pine to use UTF8 as well. With this configuration, pine can translate any email or news article from its original character set into utf8 on the fly as you expect, which then displays correctly in Absolute. However, if you try to use an editor that does not understand UTF8, you're bound to fail. The editor will not get the cursor positioning correct because it is expecting one byte per character, where utf8 can be more than one byte per character. The editor built into pine (pico) should handle UTF8 just fine, though.

What version of Pine are you using?

The keyboard input configuration in Windows does not need to be understood by Joe. AbsoluteTelnet understands this part. Absolute will convert the keystrokes according to whatever translation is in effect in Absolute (utf8 for example). According to the Joe website, joe *does* understand UTF8, and is not just a simple byte for byte editor.

The short answer is that there is a bit of configuration to get everything set correctly to UTF8, but once it's done, translation of your posts should be done for you on the fly. Otherwise, you be restricted to entering 8859-1 code points by punching in the decimal values. Not very pleasant, I imagine.

If you need any help with any portion of this, let me know.

Brian

In order to enter high-bit-set characters, I type Ctrl- (that's Control-backslash) and a key from the lower-128 -- I have a table -- and that creates the upper-128 character

Is this a feature of Joe? I'm trying to figure out why turning off VTKeypad would have broken the old way you do this. Normally, in Windows, you hold down the alt-key and type a number to create characters that are not on your keyboard or switch keyboard types via the language bar. For example, if I wanted to use UTF8 character set and enter a Euro symbol, which is not on my keyboard, I could switch Absolute over to UTF8 and then enter 'alt-0128' (using the numbers on the keypad). I could switch Absolute over to 8859-15 and do the same (alt-0128). On the screen, these two things will look the same. However, if you're using editors, the editor has to know the encoding to account for the difference between the character count and the byte count. In utf8, the Euro is three bytes, while in 8859-15, it is one. Here's a table of other characters you can type this way....

<broken link removed>

If you try to type a character that is not available in the current character set, you just see a '?'. For example, if you try to type a Euro (alt-0128) using 8859-1, it won't work because the Euro does not exist in 8859-1. You could do all of the above using a Windows keyboard layout that actually has a Euro on it as well. One thing you have to try to remember is to separate in your mind the way you type a particular character (the input method) with any particular character set. In the example above, I used alt-0128 to enter a Euro symbol. THis is not an Absolute-Telnet thing, this is a WINDOWS thing. You can do it in any app. Try doing it in notepad, for example... Absolute translates any character you type, including the special alt-XXXX characters according to the character set translation specified in Options->Properties->Appearance.

You have two responses here with a lot of material, so I will not try to respond to everything at once. However, I will set Pine (4.64, which I think was the last version ever released) and AT to UTF-8. I have done that in the past with satisfactory results on a per-message basis. It is just tedious to do it message by message. (I will also have to set some environment variables, but that is trivial in the login files.) I know when I have done that in the past, if I read a post in any other character set than UTF-8, Pine puts up an informational message.

I have not looked at the joe site in a long time. The version my ISP supplies is 3.3. At one point they offered a later version, but the results were so bad and there was so much negative feedback that they regressed to 3.3. So far as I have ever understood, that version is not UTF-8 compatible, even if later versions are. (Whether I could build a later version of joe in my personal disk allotment and PATH I don't know, as I am not an experienced Unix-type programmer, and the make files may try to install it globally, which, obviously, I cannot do.)

Entering 8-bit characters in joe 3.3 is actually quite simple. In fact, I find it *simpler* than mucking about with Alt-xxxx sequences (particularly considering that I habitually keep NumLock off), and I have never even tried that with joe. Suppose I have AT's display set to ISO 8859-1 and I want to enter lowercase e-acute in joe 3.3. All I do is press Ctrl-backslash followed by 'i' (no quotes). That's all. Fewer strokes than jibbering around with M$ Alt sequences, and I can do what I want with the NumLock key. Also, I looked at the Windows Control Panel (I have SP Pro SP3), and I am not sure what you are referring to. I confess that I am a simple user, not a geek.

Interim note: I set things up for AT, Pine, and tcsh for UTF-8. I can no longer use the joe control sequences to see Latin-1 characters correctly with no problem. I can only use the M$ Alt-xxx sequences, which to me are a royal pain. (For regular use I would have to create and print off a table.) However, I need to be able to type *non-Latin-1* characters (specifically the Esperanto accented letters), and I do not know how to do that. As of this writing I have not yet received an adequate message/post to test that. That is why I am using a non-AT client to handle those messages. (Also, whether we like it or not, more and more legitimate messages are coming with allowable HTML formatting and graphics, and so far as I understand, AT is not usable for them.)

I will keep you posted. As always, thank you for your support.

Paul Bartlett

An update. In order to handle the most frequent situation I have to deal with, I have had to go back to setting the 'VT Keypad' option and keeping AT and Pine set to ISO-8859-1. When I tried UTF-8 (including setting host-side environment variables, which is trivial) on both ends, then if someone sent a message in 8859-1 with characters in the upper range, they simply disappeared from the display. I don't know whether it makes any difference, but the terminal type in AT is set to 'xterm', and in my host-side login I have 'setenv TERM "xterm-color"'. This allows me to use the color features in Pine and joe 3.3 Lately I have not had a recurrence of the original problem which caused me to open this thread.

Can you forward the message to me? I also read my email in pine, so perhaps I can help work out the issues with this particular email. You can edit out any parts of the email that may be personal as long as you leave the problem text in there.

Brian

I will forward a message "under separate cover." It is a mailing list post, so there is nothing especially personal about it. This particular message has some text in French, and the accented letters disappear when I switch over to UTF-8. Again please note that the version of Pine I use is the one supplied by my ISP, and I have no control over it.

Paul Bartlett

Paul,

I got your message and I've verified the behavior you're getting using un-patched pine. Really, the only way to fix this is to either apply the pine utf8 patch or install 'alpine', the pine replacement, which handles utf8 beautifully. Without tools that properly convert to/from UTF8, you'll be stuck trying to match the Absolute character set translation to the character set of a given message, which is bound to be problematic.

I did try to reproduce your problem typing in accented characters in joe, but did not have the same problem you did. For my first attempt, I left 'VT Keypad' enabled and hit ctrl-backslash (the word Meta-) appeared at the bottom of my screen, then I hit 'a'. A reverse-colored 'a' appeared on the screen. For my second attempt, I did the same thing with 'vt keypad' disabled. I get the same thing, with the reverse-colored 'a' appearing. If I then exit joe and cat the file, I see 'áá'

This type of text entry leaves a lot to be desired. While the accented 'á' is easy enough, in order to get an accented 'í', I have to enter ctrl-backslash followed by an 'm'!?!? This shows as a reverse-colored 'm' in joe, but shows as 'í' (i-acute) when you cat the file.

There is a better way....

First, choose a reasonable LANG variable. It doesn't really matter, but be *sure* that the character set in Absolute matches what you use for the LANG variable.

If you need to stick with ISO-8859-1 for historical purposes, use
export LANG=en_US.ISO-8859-1

A better choice would be UTF8 if all of your software supports it
export LANG=en_US.UTF-8

With a problem LANG variable set (and matching AbsoluteTelnet/SSH), you should at least be able to see the correct characters as you type them. Now we'll add the us-international keyboard so you can type accented characters using modern input methods

1. Go to the control panel in windows
2. Double-click regional settings
3. Click the 'languages' tab
4. Click the 'details' button under 'text services and input languages'
5. Click on 'english' and click the 'add' button
6. check the 'keyboard' checkbox and choose the 'United States-International' keyboard
7. Click 'ok' all the way out.

On your task bar, you should now see the language bar. You can switch back and forth between the standard 'us' keyboard and the US International keyboard. The difference is that using the international keyboard makes typing accented characters much easier. For exable, to type a-acute , you type a single-quote followed by an 'a' to get á to get a-grave, you type a single back-quote (`) followed by an a to get à. The same goes for é, è, ú, ù, etc..... Very easy and intuitive, without the need for a code chart or lookup table.

Hopefully, some of this will help. Take a few minutes to try the international keyboard and get back to me with what still does or does not work.

bpence wrote:

Paul,

I got your message and I've verified the behavior you're getting using un-patched pine. Really, the only way to fix this is to either apply the pine utf8 patch or install 'alpine', the pine replacement, which handles utf8 beautifully. Without tools that properly convert to/from UTF8, you'll be stuck trying to match the Absolute character set translation to the character set of a given message, which is bound to be problematic.

My ISP does provide Alpine (I think), but I have been using Pine for so many years now I never tried to make a transition. I suppose that would mean setting up a whole new .rc file, which could be a bother.

I did try to reproduce your problem typing in accented characters in joe, but did not have the same problem you did. For my first attempt, I left 'VT Keypad' enabled and hit ctrl-backslash (the word Meta-) appeared at the bottom of my screen, then I hit 'a'. A reverse-colored 'a' appeared on the screen. For my second attempt, I did the same thing with 'vt keypad' disabled. I get the same thing, with the reverse-colored 'a' appearing. If I then exit joe and cat the file, I see 'áá'

This type of text entry leaves a lot to be desired. While the accented 'á' is easy enough, in order to get an accented 'í', I have to enter ctrl-backslash followed by an 'm'!?!? This shows as a reverse-colored 'm' in joe, but shows as 'í' (i-acute) when you cat the file.

As I recall, at least in the version of joe my ISP provides (3.3), what you are describing is the default behavior of joe. The editor is personally customizable with a $HOME/.joerc file, and the way I have mine set up is that when I enter ctrl-backslash and a letter, I get the high-register character directly without any reverse video. The accented letter is right there on the display in the edit space. Unfortunately, it has been so long since I set up my .joerc file that I no longer remember what the necessary setting is. If one enters the same set of accented letters regularly, so that one remembers them, entry is actually simpler and quicker than with the alt-xxxx cumbersomeness of Windows.

Also, our respective setups may not be quite the same. I am using tcsh for my shell, and I am not enough of a Unix guru to know whether the shell makes any difference.

There is a better way....

First, choose a reasonable LANG variable. It doesn't really matter, but be *sure* that the character set in Absolute matches what you use for the LANG variable.

If you need to stick with ISO-8859-1 for historical purposes, use
export LANG=en_US.ISO-8859-1

A better choice would be UTF8 if all of your software supports it
export LANG=en_US.UTF-8

So far as I understand, 'export' is not a tcsh directive. It has 'set' and 'setenv'. When I have wanted to use UTF-8, I have a macro that issues

setenv LC_CTYPE en_US.UTF-8

setenv LANG en_US.UTF-8

So far as I know, I need both of them.

With a problem LANG variable set (and matching AbsoluteTelnet/SSH), you should at least be able to see the correct characters as you type them. Now we'll add the us-international keyboard so you can type accented characters using modern input methods

...trim instructions...

On your task bar, you should now see the language bar. You can switch back and forth between the standard 'us' keyboard and the US International keyboard. The difference is that using the international keyboard makes typing accented characters much easier. For exable, to type a-acute , you type a single-quote followed by an 'a' to get á to get a-grave, you type a single back-quote (`) followed by an a to get à. The same goes for é, è, ú, ù, etc..... Very easy and intuitive, without the need for a code chart or lookup table.

I can try that. However, does it enable entry of DBCS Unicode characters, which is what I really need? For some of that, I have had to go outside AT/Pine/joe entirely and use a different email/news agent (Opera) that eats UTF-8 for breakfast, together with a special keyboard driver.

Hopefully, some of this will help. Take a few minutes to try the international keyboard and get back to me with what still does or does not work.

(Later) I tried it, but it did not seem to work at all well with joe, and I could not figure out how to get vowel letters with umlaut (the two little dots). It seems clumsier than the method I have been using.

Thanks again.
Paul Bartlett

My ISP does provide Alpine (I think), but I have been using Pine for so many years now I never tried to make a transition. I suppose that would mean setting up a whole new .rc file, which could be a bother.

I installed and used alpine for the first time yesterday. It looks and works exactly like pine, including using the same .rc file. I think you'll find the transition to be quite easy. Alpine came from pine, so I think you'll be ok with it, assuming it is available on your system.

If one enters the same set of accented letters regularly, so that one remembers them, entry is actually simpler and quicker than with the alt-xxxx cumbersomeness of Windows.

Right. I no longer recommend the alt-xxx stuff, because that is just as hard to remember as control-backslash-letter. The international keyboard is the way to go.

So far as I understand, 'export' is not a tcsh directive. It has 'set' and 'setenv'. When I have wanted to use UTF-8, I have a macro that issues...

Yes, that is just a difference in shell. No need to worry.

I can try that. However, does it enable entry of DBCS Unicode characters, which is what I really need? For some of that, I have had to go outside AT/Pine/joe entirely and use a different email/news agent (Opera) that eats UTF-8 for breakfast, together with a special keyboard driver.

Absolute can send any Unicode character to the server (provided you can type it), including Chinese, Japanese, Hebrew, Arabic, etc.... You'll have to give me a specific example if you need some help. Remember, though, that the character *has* to exist in the current character set, which is what makes UTF-8 so appealing. If you have Absolute set at 8859-1, you can't type Chinese characters, for example.

(Later) I tried it, but it did not seem to work at all well with joe, and I could not figure out how to get vowel letters with umlaut (the two little dots). It seems clumsier than the method I have been using.

You'll have to be a bit more specific about what does and doesn't work. On the international keyboard, the umlaut can be typed by preceding the character with a double-quote character("). So, when you type "a, it appears as ä. Same with ö, ü, ë, Ö, Ü, Ë, etc...

So,

I have three ways for you to type an iso8859-1 character into joe. First is the joe control-backslash method, which seems to work fine for me, even with vt keypad disabled. Second is the Windows alt-xxxx method. Third is the international keyboard method.

Example: í LATIN SMALL LETTER I WITH ACUTE

1. control-backslash-m
2. alt-0237
3. 'i (using US international keyboard)

Example: ö LATIN SMALL LETTER O WITH DIAERESIS

1. control-backslash-v
2. alt-0246
3. "o

Now you may have your opinions, but I can tell you which one I think is less clumsy. Number 3! The only benefit to number 1 is that it will work even without any special support from the operating system. However, it is no more intuitive than method 2. In fact, they're very similar. With method 2, you're typing the decimal value of the ISO8859-1 character directly, where in method 1, you're turing off the high-bit and typing the equivalent ascii character in that position.

Methods 2 and 3 will continue to work, however, event with Absolute running in UTF8 mode. Method 1 will not. With method 3, even I can type the accented characters without any need for a lookup table or cheat sheet.

All three of these methods seem to work fine for me using AT with joe and alpine (and even pine if you stick with ISO-8859-1)

If you're still having problems, please give me specific instructions on how to reproduce, including environment variables, application settings, and keystrokes I can use to reproduce the issues.

Brian

It may be a day or two before I can follow up on everything needful, but I did do a little poking around. My ISP does have Alpine 2.00 installed, which I think was the last version of Alpine before the University of Washington abandoned Pine and Alpine. I haven't really done anything with it yet, but I did notice in one of the help files that internally it tries to use UTF-8. I have a monospaced UTF-8 font installed with enough of the Unicode glyphs for what I need, as both AT and Pine/Alpine seem to assume monospaced fonts.

Also, they have joe 3.7 installed, although it is not the default. This version is UTF-8-aware if certain environment variables and internal parameters are set. I seem to recall now that at one time they had installed v3.6, but there were problems and a lot of adverse feedback, so they dropped it. I was not aware that they had installed v3.7 as a option. Completely apart from international keyboards, alt-xxxx sequences, ctl-backslash-letter, and all the rest, I have a third-party keyboard driver I can use which allows easy entry of the Unicode characters that I need, at least from a hasty test with joe-3.7.

Again, I do not have time just now to follow up on all this, but it might be that with Alpine, joe-3.7, and an AT setting, I might be able to get what I need. Ironically, the problem for which I first opened this thread has not recurred.

Paul Bartlett

Status Update from my post of last night.

I took Brian's suggestion and tried Alpine 2.00. First I set some shell environment variables for UTF-8, which Alpine seems to read and make use of. Then I set AT to use UTF-8 with the monospaced Courier font I have installed which contains at least most of the Unicode glyphs I need. Started Alpine. It looks like I might need to do some tweaking (particularly the color scheme), but otherwise most things seemed to look more or less the way they did with Pine. I received one mailing list post in three languages (even though I don't actually read all three). There were three sets of characters in it: English-Latin, Polish-Latin, and Russian-Cyrillic. As nearly as I could tell everything was there. (I don't know Russian but am familiar with the alphabet.) So there is some progress.

As always, thanks.
Paul Bartlett