Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Detect encoding and decode text response #256

Merged
merged 5 commits into from
Feb 15, 2018
Merged

Detect encoding and decode text response #256

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
25378d6
Detect encoding and decode text response
messense Feb 10, 2018
f5d00b6
Try to get encoding from Content-Type header
messense Feb 10, 2018
2337017
Remove uchardet encoding detection for now
messense Feb 11, 2018
19c57f6
Add non utf-8 test case for Response::text()
messense Feb 11, 2018
8c08daa
Reduce copies
messense Feb 15, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Try to get encoding from Content-Type header
  • Loading branch information
@messense
messense committed Feb 10, 2018
commit f5d00b640bd498d0efc95d9ed5743d4d743c5a77
10 changes: 9 additions & 1 deletion src/response.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,15 @@ impl Response {
.unwrap_or(0);
let mut content = Vec::with_capacity(len as usize);
self.read_to_end(&mut content).map_err(::error::from)?;
let encoding_name = uchardet::detect_encoding_name(&content).unwrap_or_else(|_| "utf-8".to_string());
let encoding_name = self.headers().get::<::header::ContentType>()
.and_then(|content_type| {
content_type.get_param("charset")
.map(|charset| charset.as_str().to_string())
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this needs to copy the string, and so could just be charset.as_str().

})
.unwrap_or_else(|| {
uchardet::detect_encoding_name(&content)
.unwrap_or_else(|_| "utf-8".to_string())
});
let encoding = Encoding::for_label(encoding_name.as_bytes()).unwrap_or(UTF_8);
let (text, _, _) = encoding.decode(&content);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure about this.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like decode returns a Cow<str>, since it may have detected that the bytes were valid UTF-8 and didn't need to do any copying. So, we can handle the Cow if it is Cow::Borrowed, that means we don't need to make a new copy, since the bytes in content were valid! Eliminating this copy is a bigger deal depending on how big the body was.

So, seems this could be handled like so:

// a block because of borrow checker
{
    let (text, _, _) = encoding.decode(&content);
    match text {
        Cow::Owned(s) => return Ok(s),
        _ => (),
    }
}
unsafe {
    // decoding returned Cow::Borrowed, meaning these bytes
    // are already valid utf8
    Ok(String::from_utf8_unchecked(content))
}

Ok(text.to_string())
Expand Down