Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Detect encoding and decode text response #256

Merged
merged 5 commits into from
Feb 15, 2018
Merged

Detect encoding and decode text response #256

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
25378d6
Detect encoding and decode text response
messense Feb 10, 2018
f5d00b6
Try to get encoding from Content-Type header
messense Feb 10, 2018
2337017
Remove uchardet encoding detection for now
messense Feb 11, 2018
19c57f6
Add non utf-8 test case for Response::text()
messense Feb 11, 2018
8c08daa
Reduce copies
messense Feb 15, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove uchardet encoding detection for now
  • Loading branch information
@messense
messense committed Feb 11, 2018
commit 23370177679aa7365d0542495e7a241b0ef8c4f3
1 change: 0 additions & 1 deletion Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ tokio-tls = "0.1"
url = "1.2"
uuid = { version = "0.5", features = ["v4"] }
hyper-proxy = "0.4.0"
uchardet = "2.0"

[dev-dependencies]
env_logger = "0.5"
Expand Down
1 change: 0 additions & 1 deletion src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,6 @@ extern crate tokio_io;
extern crate tokio_tls;
extern crate url;
extern crate uuid;
extern crate uchardet;

pub use hyper::header;
pub use hyper::mime;
Expand Down
6 changes: 1 addition & 5 deletions src/response.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ use encoding_rs::{Encoding, UTF_8};
use futures::{Async, Poll, Stream};
use serde::de::DeserializeOwned;
use serde_json;
use uchardet;

use client::KeepCoreThreadAlive;
use header::Headers;
Expand Down Expand Up @@ -189,10 +188,7 @@ impl Response {
content_type.get_param("charset")
.map(|charset| charset.as_str().to_string())
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this needs to copy the string, and so could just be charset.as_str().

})
.unwrap_or_else(|| {
uchardet::detect_encoding_name(&content)
.unwrap_or_else(|_| "utf-8".to_string())
});
.unwrap_or_else(|| "utf-8".to_string());
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With no copy of the string above, this can just be .unwrap_or("utf-8").

let encoding = Encoding::for_label(encoding_name.as_bytes()).unwrap_or(UTF_8);
let (text, _, _) = encoding.decode(&content);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure about this.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like decode returns a Cow<str>, since it may have detected that the bytes were valid UTF-8 and didn't need to do any copying. So, we can handle the Cow if it is Cow::Borrowed, that means we don't need to make a new copy, since the bytes in content were valid! Eliminating this copy is a bigger deal depending on how big the body was.

So, seems this could be handled like so:

// a block because of borrow checker
{
    let (text, _, _) = encoding.decode(&content);
    match text {
        Cow::Owned(s) => return Ok(s),
        _ => (),
    }
}
unsafe {
    // decoding returned Cow::Borrowed, meaning these bytes
    // are already valid utf8
    Ok(String::from_utf8_unchecked(content))
}

Ok(text.to_string())
Expand Down