Problem reading accent letters in database #414

SveinBerger · 2015-09-14T10:36:51Z

Hi,
Thanks for a great product.

My Name is Svein Berger (mr.) I'm a Norwegian software developer.
First I want to apologize for my limited knowledge of English. So please try to understand my note.

We are using an technical application that saves data to a SQLite database.
The first version of the program was written in Delphi. Later I rewrote the program in C#.
The character set back then was Windows western europe (same as CP1252)
When I rewrote the program to C# i kept the old character set. That was because it was to complicated to convert the data to UTF-8.

As a Norwegian (it applies to most countries in Europe), we use letters that are outside A..Z like øæå ö ü etc.
At the time we startet to use our database UTF-8 was not the common character sets. The transition to UTF-8 is no problem for those who only use the letters A..Z. But for us who live in countries that have an extended alphabet, the transition to UTF-8 is a challenge for some older systems. The letters A..Z kept the same places in the new character set, but the letters that we use was "shuffled around".
So the result it that when we use a program, liker yours great app, the letters øæå is displayed wrong.
An example: Pressure test is in Norwegian Trykkprøving. But it is like this Trykkpr�ving in DB Browser for SQLite. For all charaters ØÆÅ øæå is displayed like �. Not very nice.

Pleas see attatched screen shot.

If I had written the program to day, I would of course used UFT-8, and everyone would be happy.

I fully understand that you do not add an additional interpretations of the character set in the program. But if you have the opportunity and see this as a challenge, I would be more than happy to test the solution for you.

If I make my self unclear, please ask me and I will try my best to answer you.

Kind regards

Svein Berger
IT Manager
TTC NORGE AS
sveinb@ttc.no

justinclift · 2015-09-14T19:05:08Z

Definitely sounds like a bug. You're using version 3.7.0 of DB4S yeah? 😄

Btw - Your english is easy to understand. No problems at all. 😁

When browsing and editing a table in the Browse Data tab allow the user to change the assumed encoding of the text in the table. We'll then try to convert from this encoding to Unicode for display purposes and to convert from Unicode to this encoding for editing purposes. This way the UI and all tools will keep using Unicode while the database format is still respected. I hope the amount conversions generated this way won't affect the performance too much. See issue #414.

MKleusberg · 2015-09-14T22:35:03Z

Hei Svein!
Thanks a lot for your report - we've always tried to keep the program UTF-8 only for simplicity but I understand your issue. Since it doesn't seem to difficult to add a function which should solve your problem and which doesn't distract other users, I don't see any reason not to implement it 😃
Now, from the screenshot it looks like you're using MacOS X. Unfortunately our nightly builds currently aren't working for MacOS X but Justin (@justinclift) might be able to upload a build for you to test. For Windows (http://rp.oldsch00l.com/sqlitebrowser/) the nightly builds are running, though.

If you have a up-to-date build running you'll see a new option called 'Set encoding' added to the menu that appears when you right click the table header:

Click on it and you'll be asked to type in your encoding:

As soon as you click OK your new encoding will be applied:

A few more notes:

This setting is per table. If you have multiple tables you'll need to change the encoding for all of them. You can change all your settings in a project file, however, so you'll only need to do this once.
If you use the Execute SQL tab for quering or changing the database you'll still have encoding issues because this part of the application hasn't been changed.
I've tested browsing, editing and using the filters. I think that's all there is but I might be missing something ;)

Let me know how this works for you!

SveinBerger · 2015-09-15T07:14:54Z

Hi Martin

Wow! Thank you very much, all participants. This was quick and amazing!
I do most of my development i Windows, and I tried it there. It works great!

All my tables (aprox. 120 tables) , in two databases uses the old cp1251 format.
How do I set it to open all tables in cp1252?

And again thank you for your great support!

Regards
Svein

Add a new menu option to the Browse Data tab for changing the encoding of all tables instead of just the one table. Also check if the encoding the user typed in exists before trying to use it to prevent a crash when an invalid encoding is used. See issue #414.

MKleusberg · 2015-09-15T09:07:41Z

Hi Svein!
Currently you can set it only for each table separately, so you'd have to change the encoding 120 times 😦 But I've just added another menu option for changing the encoding of all tables. It's called 'Set encoding for all tables'. So with the next nightly build (tomorrow) you should be able to change the encoding for all tables at once. If you have multiple databases you'll still have to do this for each database, however. But I suppose this is do-able for two databases 😃
I hope this helps!

SveinBerger · 2015-09-16T06:17:26Z

Hi Martin,
Thank you again, this works fine, all tables are set correctly. And yes, there is no problem to set the encoding every time I use the program.

I Run a query in "Execute SQL". The result there is still UTF-8. Is it possible to include the selected encoding in the query tab?

You guys are awesome. Great support and fast solution.
Thank you.

justinclift · 2015-09-16T18:17:38Z

Sorry guys, I'm stuck on a Windows box for a few days. (groan)

@SveinBerger Do you use Homebrew on OSX? If so, compiling our code is extremely simple. You can cut-n-paste these commands to the command line, and out pops an .app you can run. 😉

https://github.com/sqlitebrowser/sqlitebrowser/blob/master/BUILDING.md#building-an-app-bundle

justinclift · 2015-09-16T18:17:59Z

Gah, wrong button. Sorry! 👼

SveinBerger · 2015-09-17T07:19:10Z

Hi Justin,
I do have Homebrew and followed your guide. It's working great. The encoding option is there and working.

Thank you.

justinclift · 2015-09-17T11:36:07Z

Awesome! Great work @MKleusberg. 😄

justinclift added the bug Confirmed bugs or reports that are very likely to be bugs. label Sep 14, 2015

justinclift assigned MKleusberg Sep 14, 2015

justinclift added this to the 3.8.0 - Awesome Sauce! milestone Sep 14, 2015

MKleusberg added enhancement Feature requests. fix applied and removed bug Confirmed bugs or reports that are very likely to be bugs. labels Sep 14, 2015

justinclift closed this as completed Sep 16, 2015

justinclift reopened this Sep 16, 2015

justinclift closed this as completed Sep 17, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem reading accent letters in database #414

Problem reading accent letters in database #414

SveinBerger commented Sep 14, 2015

justinclift commented Sep 14, 2015

MKleusberg commented Sep 14, 2015

SveinBerger commented Sep 15, 2015

MKleusberg commented Sep 15, 2015

SveinBerger commented Sep 16, 2015

justinclift commented Sep 16, 2015

justinclift commented Sep 16, 2015

SveinBerger commented Sep 17, 2015

justinclift commented Sep 17, 2015

Problem reading accent letters in database #414

Problem reading accent letters in database #414

Comments

SveinBerger commented Sep 14, 2015

justinclift commented Sep 14, 2015

MKleusberg commented Sep 14, 2015

SveinBerger commented Sep 15, 2015

MKleusberg commented Sep 15, 2015

SveinBerger commented Sep 16, 2015

justinclift commented Sep 16, 2015

justinclift commented Sep 16, 2015

SveinBerger commented Sep 17, 2015

justinclift commented Sep 17, 2015